2023-02-11 10:00:08,481 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b 2023-02-11 10:00:08,494 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-11 10:00:08,507 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 10:00:08,507 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023, deleteOnExit=true 2023-02-11 10:00:08,507 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 10:00:08,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/test.cache.data in system properties and HBase conf 2023-02-11 10:00:08,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 10:00:08,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/hadoop.log.dir in system properties and HBase conf 2023-02-11 10:00:08,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 10:00:08,510 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 10:00:08,510 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 10:00:08,625 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 10:00:09,015 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 10:00:09,019 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 10:00:09,020 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 10:00:09,020 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 10:00:09,020 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 10:00:09,021 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 10:00:09,021 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 10:00:09,021 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 10:00:09,022 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 10:00:09,022 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 10:00:09,022 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/nfs.dump.dir in system properties and HBase conf 2023-02-11 10:00:09,023 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/java.io.tmpdir in system properties and HBase conf 2023-02-11 10:00:09,023 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 10:00:09,024 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 10:00:09,024 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 10:00:09,531 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 10:00:09,536 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 10:00:09,845 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 10:00:10,043 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 10:00:10,059 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 10:00:10,103 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 10:00:10,135 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/java.io.tmpdir/Jetty_localhost_39095_hdfs____vyzlqo/webapp 2023-02-11 10:00:10,278 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39095 2023-02-11 10:00:10,293 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 10:00:10,293 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 10:00:10,768 WARN [Listener at localhost/40183] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 10:00:10,831 WARN [Listener at localhost/40183] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 10:00:10,848 WARN [Listener at localhost/40183] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 10:00:10,854 INFO [Listener at localhost/40183] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 10:00:10,858 INFO [Listener at localhost/40183] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/java.io.tmpdir/Jetty_localhost_37511_datanode____.rxf1ig/webapp 2023-02-11 10:00:10,958 INFO [Listener at localhost/40183] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37511 2023-02-11 10:00:11,258 WARN [Listener at localhost/36575] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 10:00:11,271 WARN [Listener at localhost/36575] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 10:00:11,274 WARN [Listener at localhost/36575] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 10:00:11,277 INFO [Listener at localhost/36575] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 10:00:11,282 INFO [Listener at localhost/36575] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/java.io.tmpdir/Jetty_localhost_41733_datanode____.2harj3/webapp 2023-02-11 10:00:11,393 INFO [Listener at localhost/36575] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41733 2023-02-11 10:00:11,404 WARN [Listener at localhost/42381] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 10:00:11,425 WARN [Listener at localhost/42381] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 10:00:11,428 WARN [Listener at localhost/42381] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 10:00:11,430 INFO [Listener at localhost/42381] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 10:00:11,437 INFO [Listener at localhost/42381] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/java.io.tmpdir/Jetty_localhost_35195_datanode____g0wb0y/webapp 2023-02-11 10:00:11,574 INFO [Listener at localhost/42381] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35195 2023-02-11 10:00:11,587 WARN [Listener at localhost/42305] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 10:00:11,732 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf47858b19d33f570: Processing first storage report for DS-1e74013c-8276-4cd7-90f9-1caea053afca from datanode c54c5566-4672-48cb-89c2-ee4576f23616 2023-02-11 10:00:11,734 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf47858b19d33f570: from storage DS-1e74013c-8276-4cd7-90f9-1caea053afca node DatanodeRegistration(127.0.0.1:41641, datanodeUuid=c54c5566-4672-48cb-89c2-ee4576f23616, infoPort=39515, infoSecurePort=0, ipcPort=42381, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,734 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfc0809674aa7a1ad: Processing first storage report for DS-0b814228-9c59-47e9-b72f-6aeadbd920be from datanode 1b1dcb87-4b37-427d-8dc6-8c93c312eb72 2023-02-11 10:00:11,734 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfc0809674aa7a1ad: from storage DS-0b814228-9c59-47e9-b72f-6aeadbd920be node DatanodeRegistration(127.0.0.1:44103, datanodeUuid=1b1dcb87-4b37-427d-8dc6-8c93c312eb72, infoPort=39623, infoSecurePort=0, ipcPort=42305, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,734 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4013309d9152d9ed: Processing first storage report for DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7 from datanode e84cdfd9-aa67-445a-9a90-f8380b7333c7 2023-02-11 10:00:11,734 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4013309d9152d9ed: from storage DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7 node DatanodeRegistration(127.0.0.1:45127, datanodeUuid=e84cdfd9-aa67-445a-9a90-f8380b7333c7, infoPort=45307, infoSecurePort=0, ipcPort=36575, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf47858b19d33f570: Processing first storage report for DS-30e6e3a6-9540-4305-bdcb-72d5219986d3 from datanode c54c5566-4672-48cb-89c2-ee4576f23616 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf47858b19d33f570: from storage DS-30e6e3a6-9540-4305-bdcb-72d5219986d3 node DatanodeRegistration(127.0.0.1:41641, datanodeUuid=c54c5566-4672-48cb-89c2-ee4576f23616, infoPort=39515, infoSecurePort=0, ipcPort=42381, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xfc0809674aa7a1ad: Processing first storage report for DS-f7f8306a-7631-44f0-bdda-1861f07c66c7 from datanode 1b1dcb87-4b37-427d-8dc6-8c93c312eb72 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xfc0809674aa7a1ad: from storage DS-f7f8306a-7631-44f0-bdda-1861f07c66c7 node DatanodeRegistration(127.0.0.1:44103, datanodeUuid=1b1dcb87-4b37-427d-8dc6-8c93c312eb72, infoPort=39623, infoSecurePort=0, ipcPort=42305, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4013309d9152d9ed: Processing first storage report for DS-85722d99-9906-42a6-8e15-4a5e7df5bf43 from datanode e84cdfd9-aa67-445a-9a90-f8380b7333c7 2023-02-11 10:00:11,735 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4013309d9152d9ed: from storage DS-85722d99-9906-42a6-8e15-4a5e7df5bf43 node DatanodeRegistration(127.0.0.1:45127, datanodeUuid=e84cdfd9-aa67-445a-9a90-f8380b7333c7, infoPort=45307, infoSecurePort=0, ipcPort=36575, storageInfo=lv=-57;cid=testClusterID;nsid=1881242358;c=1676109609623), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 10:00:11,966 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b 2023-02-11 10:00:12,032 INFO [Listener at localhost/42305] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/zookeeper_0, clientPort=64863, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 10:00:12,049 INFO [Listener at localhost/42305] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=64863 2023-02-11 10:00:12,061 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:12,064 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:12,797 INFO [Listener at localhost/42305] util.FSUtils(479): Created version file at hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7 with version=8 2023-02-11 10:00:12,797 INFO [Listener at localhost/42305] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/hbase-staging 2023-02-11 10:00:13,143 INFO [Listener at localhost/42305] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 10:00:13,665 INFO [Listener at localhost/42305] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 10:00:13,699 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:13,700 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:13,700 INFO [Listener at localhost/42305] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 10:00:13,701 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:13,701 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 10:00:13,843 INFO [Listener at localhost/42305] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 10:00:13,917 DEBUG [Listener at localhost/42305] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 10:00:14,011 INFO [Listener at localhost/42305] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42723 2023-02-11 10:00:14,026 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,029 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,053 INFO [Listener at localhost/42305] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42723 connecting to ZooKeeper ensemble=127.0.0.1:64863 2023-02-11 10:00:14,100 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:427230x0, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 10:00:14,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:42723-0x10042d2edb60000 connected 2023-02-11 10:00:14,136 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 10:00:14,137 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:14,140 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 10:00:14,148 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42723 2023-02-11 10:00:14,149 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42723 2023-02-11 10:00:14,149 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42723 2023-02-11 10:00:14,149 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42723 2023-02-11 10:00:14,150 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42723 2023-02-11 10:00:14,155 INFO [Listener at localhost/42305] master.HMaster(439): hbase.rootdir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7, hbase.cluster.distributed=false 2023-02-11 10:00:14,245 INFO [Listener at localhost/42305] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 10:00:14,245 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,245 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,245 INFO [Listener at localhost/42305] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 10:00:14,246 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,246 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 10:00:14,251 INFO [Listener at localhost/42305] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 10:00:14,254 INFO [Listener at localhost/42305] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41193 2023-02-11 10:00:14,256 INFO [Listener at localhost/42305] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 10:00:14,262 INFO [Listener at localhost/42305] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 10:00:14,263 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,265 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,267 INFO [Listener at localhost/42305] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41193 connecting to ZooKeeper ensemble=127.0.0.1:64863 2023-02-11 10:00:14,271 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:411930x0, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 10:00:14,272 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:411930x0, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 10:00:14,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41193-0x10042d2edb60001 connected 2023-02-11 10:00:14,274 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:14,275 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 10:00:14,276 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41193 2023-02-11 10:00:14,277 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41193 2023-02-11 10:00:14,280 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41193 2023-02-11 10:00:14,280 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41193 2023-02-11 10:00:14,281 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41193 2023-02-11 10:00:14,293 INFO [Listener at localhost/42305] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 10:00:14,293 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,294 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,294 INFO [Listener at localhost/42305] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 10:00:14,295 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,295 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 10:00:14,295 INFO [Listener at localhost/42305] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 10:00:14,297 INFO [Listener at localhost/42305] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41567 2023-02-11 10:00:14,297 INFO [Listener at localhost/42305] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 10:00:14,298 INFO [Listener at localhost/42305] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 10:00:14,299 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,301 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,302 INFO [Listener at localhost/42305] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41567 connecting to ZooKeeper ensemble=127.0.0.1:64863 2023-02-11 10:00:14,306 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:415670x0, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 10:00:14,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41567-0x10042d2edb60002 connected 2023-02-11 10:00:14,307 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 10:00:14,308 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:14,309 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 10:00:14,310 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41567 2023-02-11 10:00:14,310 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41567 2023-02-11 10:00:14,314 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41567 2023-02-11 10:00:14,314 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41567 2023-02-11 10:00:14,315 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41567 2023-02-11 10:00:14,327 INFO [Listener at localhost/42305] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-11 10:00:14,327 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,327 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,327 INFO [Listener at localhost/42305] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 10:00:14,328 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 10:00:14,328 INFO [Listener at localhost/42305] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 10:00:14,328 INFO [Listener at localhost/42305] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 10:00:14,329 INFO [Listener at localhost/42305] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35443 2023-02-11 10:00:14,329 INFO [Listener at localhost/42305] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 10:00:14,332 INFO [Listener at localhost/42305] mob.MobFileCache(124): MobFileCache disabled 2023-02-11 10:00:14,333 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,335 INFO [Listener at localhost/42305] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,337 INFO [Listener at localhost/42305] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35443 connecting to ZooKeeper ensemble=127.0.0.1:64863 2023-02-11 10:00:14,340 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:354430x0, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 10:00:14,341 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:354430x0, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 10:00:14,341 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:354430x0, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:14,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35443-0x10042d2edb60003 connected 2023-02-11 10:00:14,342 DEBUG [Listener at localhost/42305] zookeeper.ZKUtil(164): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 10:00:14,346 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35443 2023-02-11 10:00:14,347 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35443 2023-02-11 10:00:14,347 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35443 2023-02-11 10:00:14,347 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35443 2023-02-11 10:00:14,348 DEBUG [Listener at localhost/42305] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35443 2023-02-11 10:00:14,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:14,358 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 10:00:14,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:14,381 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 10:00:14,381 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 10:00:14,381 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 10:00:14,381 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 10:00:14,382 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:14,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 10:00:14,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42723,1676109612965 from backup master directory 2023-02-11 10:00:14,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 10:00:14,389 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:14,389 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 10:00:14,390 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 10:00:14,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:14,393 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 10:00:14,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 10:00:14,498 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/hbase.id with ID: e9782f84-b5be-44a8-aefa-74547bbdba6d 2023-02-11 10:00:14,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 10:00:14,560 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:14,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x08a97f85 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:14,640 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e937b9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:14,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 10:00:14,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 10:00:14,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 10:00:14,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 10:00:14,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 10:00:14,688 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 10:00:14,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 10:00:14,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store-tmp 2023-02-11 10:00:14,761 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:14,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 10:00:14,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:14,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:14,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 10:00:14,762 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:14,763 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:14,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676109614761Disabling compacts and flushes for region at 1676109614762 (+1 ms)Disabling writes for close at 1676109614762Writing region close event to WAL at 1676109614763 (+1 ms)Closed at 1676109614763 2023-02-11 10:00:14,764 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/WALs/jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:14,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42723%2C1676109612965, suffix=, logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/WALs/jenkins-hbase4.apache.org,42723,1676109612965, archiveDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/oldWALs, maxLogs=10 2023-02-11 10:00:14,868 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK] 2023-02-11 10:00:14,869 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK] 2023-02-11 10:00:14,873 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK] 2023-02-11 10:00:14,882 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 10:00:14,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/WALs/jenkins-hbase4.apache.org,42723,1676109612965/jenkins-hbase4.apache.org%2C42723%2C1676109612965.1676109614800 2023-02-11 10:00:14,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK], DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK], DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK]] 2023-02-11 10:00:14,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 10:00:14,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:15,002 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,090 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,099 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 10:00:15,125 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 10:00:15,138 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:15,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,147 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 10:00:15,167 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:15,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63775833, jitterRate=-0.04966603219509125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:15,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676109615049Initializing all the Stores at 1676109615051 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676109615052 (+1 ms)Cleaning up temporary data from old regions at 1676109615154 (+102 ms)Cleaning up detritus from prior splits at 1676109615155 (+1 ms)Region opened successfully at 1676109615168 (+13 ms) 2023-02-11 10:00:15,170 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 10:00:15,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 10:00:15,190 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 10:00:15,192 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 10:00:15,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-11 10:00:15,226 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-11 10:00:15,226 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 10:00:15,252 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 10:00:15,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 10:00:15,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 10:00:15,286 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 10:00:15,288 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 10:00:15,293 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 10:00:15,300 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 10:00:15,303 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:15,308 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 10:00:15,308 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 10:00:15,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 10:00:15,327 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:15,327 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:15,327 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:15,327 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:15,327 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:15,328 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,42723,1676109612965, sessionid=0x10042d2edb60000, setting cluster-up flag (Was=false) 2023-02-11 10:00:15,346 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:15,353 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 10:00:15,355 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:15,360 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:15,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 10:00:15,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:15,371 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp 2023-02-11 10:00:15,454 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(952): ClusterId : e9782f84-b5be-44a8-aefa-74547bbdba6d 2023-02-11 10:00:15,461 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(952): ClusterId : e9782f84-b5be-44a8-aefa-74547bbdba6d 2023-02-11 10:00:15,462 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(952): ClusterId : e9782f84-b5be-44a8-aefa-74547bbdba6d 2023-02-11 10:00:15,467 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 10:00:15,467 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 10:00:15,477 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 10:00:15,482 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 10:00:15,482 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 10:00:15,482 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 10:00:15,482 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 10:00:15,482 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 10:00:15,482 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 10:00:15,497 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 10:00:15,499 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ReadOnlyZKClient(139): Connect 0x73e33b76 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:15,502 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 10:00:15,504 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ReadOnlyZKClient(139): Connect 0x4a654399 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:15,508 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 10:00:15,509 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ReadOnlyZKClient(139): Connect 0x17fbc61c to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:15,540 DEBUG [RS:2;jenkins-hbase4:35443] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d24db7a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:15,541 DEBUG [RS:2;jenkins-hbase4:35443] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@791b5006, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 10:00:15,544 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 10:00:15,544 DEBUG [RS:0;jenkins-hbase4:41193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@193c3a93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:15,544 DEBUG [RS:0;jenkins-hbase4:41193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d03ad6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 10:00:15,548 DEBUG [RS:1;jenkins-hbase4:41567] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44a2a4a5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:15,548 DEBUG [RS:1;jenkins-hbase4:41567] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@409d28e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 10:00:15,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 10:00:15,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676109645563 2023-02-11 10:00:15,565 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 10:00:15,578 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:41567 2023-02-11 10:00:15,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 10:00:15,580 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:35443 2023-02-11 10:00:15,580 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 10:00:15,581 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 10:00:15,581 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41193 2023-02-11 10:00:15,583 INFO [RS:1;jenkins-hbase4:41567] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 10:00:15,583 INFO [RS:1;jenkins-hbase4:41567] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 10:00:15,583 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 10:00:15,584 INFO [RS:2;jenkins-hbase4:35443] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 10:00:15,584 INFO [RS:2;jenkins-hbase4:35443] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 10:00:15,584 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 10:00:15,586 INFO [RS:0;jenkins-hbase4:41193] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 10:00:15,586 INFO [RS:0;jenkins-hbase4:41193] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 10:00:15,586 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 10:00:15,587 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 10:00:15,588 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42723,1676109612965 with isa=jenkins-hbase4.apache.org/172.31.14.131:41567, startcode=1676109614293 2023-02-11 10:00:15,589 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 10:00:15,590 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 10:00:15,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 10:00:15,591 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 10:00:15,598 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42723,1676109612965 with isa=jenkins-hbase4.apache.org/172.31.14.131:41193, startcode=1676109614244 2023-02-11 10:00:15,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,598 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42723,1676109612965 with isa=jenkins-hbase4.apache.org/172.31.14.131:35443, startcode=1676109614327 2023-02-11 10:00:15,600 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 10:00:15,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 10:00:15,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 10:00:15,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 10:00:15,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 10:00:15,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676109615608,5,FailOnTimeoutGroup] 2023-02-11 10:00:15,614 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676109615612,5,FailOnTimeoutGroup] 2023-02-11 10:00:15,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 10:00:15,615 DEBUG [RS:0;jenkins-hbase4:41193] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 10:00:15,616 DEBUG [RS:2;jenkins-hbase4:35443] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 10:00:15,615 DEBUG [RS:1;jenkins-hbase4:41567] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 10:00:15,616 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,617 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,674 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:15,677 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:15,678 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7 2023-02-11 10:00:15,703 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47545, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 10:00:15,703 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51777, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 10:00:15,703 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39341, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 10:00:15,729 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42723] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,729 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42723] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,741 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:15,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 10:00:15,753 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/info 2023-02-11 10:00:15,755 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7 2023-02-11 10:00:15,755 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7 2023-02-11 10:00:15,755 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7 2023-02-11 10:00:15,756 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40183 2023-02-11 10:00:15,755 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 10:00:15,755 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40183 2023-02-11 10:00:15,756 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 10:00:15,756 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40183 2023-02-11 10:00:15,756 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 10:00:15,757 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 10:00:15,758 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:15,758 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 10:00:15,761 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/rep_barrier 2023-02-11 10:00:15,762 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 10:00:15,763 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:15,763 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 10:00:15,767 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:15,767 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/table 2023-02-11 10:00:15,768 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 10:00:15,769 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:15,769 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ZKUtil(162): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,770 WARN [RS:0;jenkins-hbase4:41193] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 10:00:15,770 INFO [RS:0;jenkins-hbase4:41193] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 10:00:15,771 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ZKUtil(162): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,771 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,771 WARN [RS:2;jenkins-hbase4:35443] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 10:00:15,771 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41567,1676109614293] 2023-02-11 10:00:15,771 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ZKUtil(162): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,772 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41193,1676109614244] 2023-02-11 10:00:15,771 INFO [RS:2;jenkins-hbase4:35443] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 10:00:15,772 WARN [RS:1;jenkins-hbase4:41567] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 10:00:15,772 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,773 INFO [RS:1;jenkins-hbase4:41567] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 10:00:15,774 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740 2023-02-11 10:00:15,775 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,772 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35443,1676109614327] 2023-02-11 10:00:15,775 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740 2023-02-11 10:00:15,793 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 10:00:15,802 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ZKUtil(162): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,802 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ZKUtil(162): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,803 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ZKUtil(162): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,803 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ZKUtil(162): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,804 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ZKUtil(162): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,805 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 10:00:15,807 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ZKUtil(162): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,809 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ZKUtil(162): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,810 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ZKUtil(162): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,811 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ZKUtil(162): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,812 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:15,813 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62123090, jitterRate=-0.0742938220500946}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 10:00:15,814 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676109615742Initializing all the Stores at 1676109615743 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676109615745 (+2 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676109615746 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676109615746Cleaning up temporary data from old regions at 1676109615783 (+37 ms)Cleaning up detritus from prior splits at 1676109615785 (+2 ms)Region opened successfully at 1676109615814 (+29 ms) 2023-02-11 10:00:15,814 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 10:00:15,814 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 10:00:15,814 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 10:00:15,814 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 10:00:15,814 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 10:00:15,820 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 10:00:15,820 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676109615814Disabling compacts and flushes for region at 1676109615814Disabling writes for close at 1676109615814Writing region close event to WAL at 1676109615820 (+6 ms)Closed at 1676109615820 2023-02-11 10:00:15,820 DEBUG [RS:0;jenkins-hbase4:41193] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 10:00:15,820 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 10:00:15,823 DEBUG [RS:2;jenkins-hbase4:35443] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 10:00:15,829 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 10:00:15,829 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 10:00:15,838 INFO [RS:1;jenkins-hbase4:41567] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 10:00:15,838 INFO [RS:2;jenkins-hbase4:35443] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 10:00:15,838 INFO [RS:0;jenkins-hbase4:41193] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 10:00:15,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 10:00:15,860 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 10:00:15,864 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:15,866 INFO [RS:2;jenkins-hbase4:35443] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 10:00:15,868 INFO [RS:0;jenkins-hbase4:41193] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 10:00:15,868 INFO [RS:1;jenkins-hbase4:41567] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 10:00:15,870 INFO [RS:2;jenkins-hbase4:35443] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 10:00:15,870 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,871 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 10:00:15,871 INFO [RS:1;jenkins-hbase4:41567] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 10:00:15,871 INFO [RS:0;jenkins-hbase4:41193] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 10:00:15,872 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,872 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,873 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 10:00:15,876 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 10:00:15,883 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,883 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,883 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,883 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,883 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 10:00:15,885 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,884 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:1;jenkins-hbase4:41567] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,885 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 10:00:15,885 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 10:00:15,886 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,886 DEBUG [RS:0;jenkins-hbase4:41193] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,887 DEBUG [RS:2;jenkins-hbase4:35443] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 10:00:15,897 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,897 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,897 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,897 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,898 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,898 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,903 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,903 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,903 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,919 INFO [RS:2;jenkins-hbase4:35443] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 10:00:15,921 INFO [RS:1;jenkins-hbase4:41567] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 10:00:15,924 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35443,1676109614327-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,924 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41567,1676109614293-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,927 INFO [RS:0;jenkins-hbase4:41193] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 10:00:15,927 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41193,1676109614244-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:15,977 INFO [RS:1;jenkins-hbase4:41567] regionserver.Replication(203): jenkins-hbase4.apache.org,41567,1676109614293 started 2023-02-11 10:00:15,977 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41567,1676109614293, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41567, sessionid=0x10042d2edb60002 2023-02-11 10:00:15,983 INFO [RS:0;jenkins-hbase4:41193] regionserver.Replication(203): jenkins-hbase4.apache.org,41193,1676109614244 started 2023-02-11 10:00:15,984 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41193,1676109614244, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41193, sessionid=0x10042d2edb60001 2023-02-11 10:00:15,984 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 10:00:15,984 DEBUG [RS:0;jenkins-hbase4:41193] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,984 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41193,1676109614244' 2023-02-11 10:00:15,984 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 10:00:15,984 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 10:00:15,985 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 10:00:15,986 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 10:00:15,986 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 10:00:15,986 DEBUG [RS:0;jenkins-hbase4:41193] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:15,986 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41193,1676109614244' 2023-02-11 10:00:15,986 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 10:00:15,987 DEBUG [RS:0;jenkins-hbase4:41193] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 10:00:15,987 DEBUG [RS:0;jenkins-hbase4:41193] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 10:00:15,988 INFO [RS:0;jenkins-hbase4:41193] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 10:00:15,988 INFO [RS:0;jenkins-hbase4:41193] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 10:00:15,985 DEBUG [RS:1;jenkins-hbase4:41567] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,996 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41567,1676109614293' 2023-02-11 10:00:15,997 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 10:00:15,997 INFO [RS:2;jenkins-hbase4:35443] regionserver.Replication(203): jenkins-hbase4.apache.org,35443,1676109614327 started 2023-02-11 10:00:15,997 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35443,1676109614327, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35443, sessionid=0x10042d2edb60003 2023-02-11 10:00:15,997 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 10:00:15,998 DEBUG [RS:2;jenkins-hbase4:35443] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:15,998 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35443,1676109614327' 2023-02-11 10:00:15,998 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 10:00:15,998 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 10:00:15,998 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 10:00:15,999 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 10:00:15,999 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 10:00:15,999 DEBUG [RS:1;jenkins-hbase4:41567] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:15,999 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 10:00:15,999 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41567,1676109614293' 2023-02-11 10:00:15,999 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 10:00:15,999 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 10:00:15,999 DEBUG [RS:2;jenkins-hbase4:35443] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:16,000 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35443,1676109614327' 2023-02-11 10:00:16,000 DEBUG [RS:1;jenkins-hbase4:41567] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 10:00:16,000 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 10:00:16,000 DEBUG [RS:1;jenkins-hbase4:41567] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 10:00:16,000 DEBUG [RS:2;jenkins-hbase4:35443] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 10:00:16,001 INFO [RS:1;jenkins-hbase4:41567] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 10:00:16,001 INFO [RS:1;jenkins-hbase4:41567] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 10:00:16,002 DEBUG [RS:2;jenkins-hbase4:35443] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 10:00:16,002 INFO [RS:2;jenkins-hbase4:35443] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 10:00:16,002 INFO [RS:2;jenkins-hbase4:35443] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 10:00:16,016 DEBUG [jenkins-hbase4:42723] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 10:00:16,020 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:16,027 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:16,027 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:16,027 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:16,027 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:16,030 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41567,1676109614293, state=OPENING 2023-02-11 10:00:16,037 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 10:00:16,039 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:16,040 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 10:00:16,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:16,101 INFO [RS:0;jenkins-hbase4:41193] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41193%2C1676109614244, suffix=, logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41193,1676109614244, archiveDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs, maxLogs=32 2023-02-11 10:00:16,105 INFO [RS:2;jenkins-hbase4:35443] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35443%2C1676109614327, suffix=, logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,35443,1676109614327, archiveDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs, maxLogs=32 2023-02-11 10:00:16,106 INFO [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41567%2C1676109614293, suffix=, logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41567,1676109614293, archiveDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs, maxLogs=32 2023-02-11 10:00:16,161 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK] 2023-02-11 10:00:16,162 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK] 2023-02-11 10:00:16,163 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK] 2023-02-11 10:00:16,164 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK] 2023-02-11 10:00:16,166 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK] 2023-02-11 10:00:16,166 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK] 2023-02-11 10:00:16,167 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK] 2023-02-11 10:00:16,167 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK] 2023-02-11 10:00:16,167 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK] 2023-02-11 10:00:16,180 INFO [RS:2;jenkins-hbase4:35443] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,35443,1676109614327/jenkins-hbase4.apache.org%2C35443%2C1676109614327.1676109616108 2023-02-11 10:00:16,180 INFO [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41567,1676109614293/jenkins-hbase4.apache.org%2C41567%2C1676109614293.1676109616108 2023-02-11 10:00:16,181 DEBUG [RS:2;jenkins-hbase4:35443] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK], DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK], DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK]] 2023-02-11 10:00:16,181 INFO [RS:0;jenkins-hbase4:41193] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41193,1676109614244/jenkins-hbase4.apache.org%2C41193%2C1676109614244.1676109616107 2023-02-11 10:00:16,181 DEBUG [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK], DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK], DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK]] 2023-02-11 10:00:16,181 DEBUG [RS:0;jenkins-hbase4:41193] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK], DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK], DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK]] 2023-02-11 10:00:16,238 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:16,240 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 10:00:16,244 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33778, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 10:00:16,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 10:00:16,262 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 10:00:16,265 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41567%2C1676109614293.meta, suffix=.meta, logDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41567,1676109614293, archiveDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs, maxLogs=32 2023-02-11 10:00:16,290 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK] 2023-02-11 10:00:16,293 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK] 2023-02-11 10:00:16,296 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK] 2023-02-11 10:00:16,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/WALs/jenkins-hbase4.apache.org,41567,1676109614293/jenkins-hbase4.apache.org%2C41567%2C1676109614293.meta.1676109616267.meta 2023-02-11 10:00:16,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45127,DS-ca0e251c-d4df-4f60-9e8f-e2a8b2fe3bf7,DISK], DatanodeInfoWithStorage[127.0.0.1:44103,DS-0b814228-9c59-47e9-b72f-6aeadbd920be,DISK], DatanodeInfoWithStorage[127.0.0.1:41641,DS-1e74013c-8276-4cd7-90f9-1caea053afca,DISK]] 2023-02-11 10:00:16,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 10:00:16,313 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 10:00:16,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 10:00:16,335 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 10:00:16,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 10:00:16,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:16,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 10:00:16,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 10:00:16,343 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 10:00:16,345 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/info 2023-02-11 10:00:16,345 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/info 2023-02-11 10:00:16,346 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 10:00:16,347 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:16,347 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 10:00:16,349 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/rep_barrier 2023-02-11 10:00:16,349 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/rep_barrier 2023-02-11 10:00:16,350 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 10:00:16,351 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:16,351 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 10:00:16,352 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/table 2023-02-11 10:00:16,352 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/table 2023-02-11 10:00:16,353 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 10:00:16,354 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:16,355 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740 2023-02-11 10:00:16,358 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740 2023-02-11 10:00:16,361 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 10:00:16,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 10:00:16,366 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65483861, jitterRate=-0.024214431643486023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 10:00:16,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676109616340Writing region info on filesystem at 1676109616340Initializing all the Stores at 1676109616342 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676109616342Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676109616343 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676109616343Cleaning up temporary data from old regions at 1676109616359 (+16 ms)Cleaning up detritus from prior splits at 1676109616360 (+1 ms)Running coprocessor post-open hooks at 1676109616367 (+7 ms)Region opened successfully at 1676109616367 2023-02-11 10:00:16,376 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676109616230 2023-02-11 10:00:16,393 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 10:00:16,394 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 10:00:16,395 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41567,1676109614293, state=OPEN 2023-02-11 10:00:16,397 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 10:00:16,397 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 10:00:16,402 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 10:00:16,402 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41567,1676109614293 in 354 msec 2023-02-11 10:00:16,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 10:00:16,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 563 msec 2023-02-11 10:00:16,415 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 990 msec 2023-02-11 10:00:16,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676109614376, completionTime=-1 2023-02-11 10:00:16,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 10:00:16,415 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 10:00:16,478 DEBUG [hconnection-0x3d381091-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:16,481 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:16,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 10:00:16,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676109676499 2023-02-11 10:00:16,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676109736499 2023-02-11 10:00:16,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 83 msec 2023-02-11 10:00:16,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42723,1676109612965-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:16,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42723,1676109612965-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:16,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42723,1676109612965-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:16,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42723, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:16,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 10:00:16,537 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 10:00:16,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 10:00:16,548 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 10:00:16,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 10:00:16,560 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 10:00:16,563 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 10:00:16,585 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:16,587 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1 empty. 2023-02-11 10:00:16,588 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:16,588 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 10:00:16,628 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:16,630 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => f5663c8b8537d642d1212d0ff668c8a1, NAME => 'hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:16,658 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:16,658 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing f5663c8b8537d642d1212d0ff668c8a1, disabling compactions & flushes 2023-02-11 10:00:16,658 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:16,658 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:16,658 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. after waiting 0 ms 2023-02-11 10:00:16,659 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:16,659 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:16,659 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for f5663c8b8537d642d1212d0ff668c8a1: Waiting for close lock at 1676109616658Disabling compacts and flushes for region at 1676109616658Disabling writes for close at 1676109616659 (+1 ms)Writing region close event to WAL at 1676109616659Closed at 1676109616659 2023-02-11 10:00:16,664 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 10:00:16,680 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676109616667"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109616667"}]},"ts":"1676109616667"} 2023-02-11 10:00:16,707 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 10:00:16,709 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 10:00:16,714 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109616710"}]},"ts":"1676109616710"} 2023-02-11 10:00:16,718 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 10:00:16,722 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:16,723 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:16,723 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:16,723 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:16,723 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:16,728 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f5663c8b8537d642d1212d0ff668c8a1, ASSIGN}] 2023-02-11 10:00:16,732 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f5663c8b8537d642d1212d0ff668c8a1, ASSIGN 2023-02-11 10:00:16,734 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=f5663c8b8537d642d1212d0ff668c8a1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:16,886 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 10:00:16,888 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f5663c8b8537d642d1212d0ff668c8a1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:16,888 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676109616887"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109616887"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109616887"}]},"ts":"1676109616887"} 2023-02-11 10:00:16,893 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure f5663c8b8537d642d1212d0ff668c8a1, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:17,057 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:17,058 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f5663c8b8537d642d1212d0ff668c8a1, NAME => 'hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.', STARTKEY => '', ENDKEY => ''} 2023-02-11 10:00:17,059 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,059 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,059 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,059 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,061 INFO [StoreOpener-f5663c8b8537d642d1212d0ff668c8a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,063 DEBUG [StoreOpener-f5663c8b8537d642d1212d0ff668c8a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/info 2023-02-11 10:00:17,063 DEBUG [StoreOpener-f5663c8b8537d642d1212d0ff668c8a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/info 2023-02-11 10:00:17,064 INFO [StoreOpener-f5663c8b8537d642d1212d0ff668c8a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f5663c8b8537d642d1212d0ff668c8a1 columnFamilyName info 2023-02-11 10:00:17,065 INFO [StoreOpener-f5663c8b8537d642d1212d0ff668c8a1-1] regionserver.HStore(310): Store=f5663c8b8537d642d1212d0ff668c8a1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:17,066 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,067 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,071 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:17,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:17,074 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f5663c8b8537d642d1212d0ff668c8a1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73938654, jitterRate=0.10177180171012878}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:17,074 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f5663c8b8537d642d1212d0ff668c8a1: Running coprocessor pre-open hook at 1676109617059Writing region info on filesystem at 1676109617059Initializing all the Stores at 1676109617061 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676109617061Cleaning up temporary data from old regions at 1676109617068 (+7 ms)Cleaning up detritus from prior splits at 1676109617068Running coprocessor post-open hooks at 1676109617074 (+6 ms)Region opened successfully at 1676109617074 2023-02-11 10:00:17,080 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1., pid=6, masterSystemTime=1676109617048 2023-02-11 10:00:17,083 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:17,083 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:17,084 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f5663c8b8537d642d1212d0ff668c8a1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:17,085 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676109617084"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109617084"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109617084"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109617084"}]},"ts":"1676109617084"} 2023-02-11 10:00:17,091 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 10:00:17,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure f5663c8b8537d642d1212d0ff668c8a1, server=jenkins-hbase4.apache.org,41567,1676109614293 in 195 msec 2023-02-11 10:00:17,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 10:00:17,096 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=f5663c8b8537d642d1212d0ff668c8a1, ASSIGN in 363 msec 2023-02-11 10:00:17,097 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 10:00:17,097 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109617097"}]},"ts":"1676109617097"} 2023-02-11 10:00:17,099 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 10:00:17,102 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 10:00:17,105 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 553 msec 2023-02-11 10:00:17,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 10:00:17,165 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 10:00:17,165 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:17,196 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 10:00:17,210 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 10:00:17,215 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 26 msec 2023-02-11 10:00:17,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 10:00:17,229 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 10:00:17,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-02-11 10:00:17,243 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 10:00:17,246 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 10:00:17,248 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.858sec 2023-02-11 10:00:17,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 10:00:17,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 10:00:17,252 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 10:00:17,253 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42723,1676109612965-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 10:00:17,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42723,1676109612965-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 10:00:17,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 10:00:17,272 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x297db6e1 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:17,276 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c633018, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:17,289 DEBUG [hconnection-0x6de71207-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:17,300 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:17,310 INFO [Listener at localhost/42305] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:17,341 INFO [Listener at localhost/42305] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=359, OpenFileDescriptor=691, MaxFileDescriptor=60000, SystemLoadAverage=193, ProcessCount=167, AvailableMemoryMB=3285 2023-02-11 10:00:17,359 DEBUG [Listener at localhost/42305] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 10:00:17,362 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47014, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 10:00:17,377 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 10:00:17,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:17,382 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 10:00:17,384 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 10:00:17,386 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676109617344" procId is: 9 2023-02-11 10:00:17,394 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:17,394 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,394 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:17,395 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,395 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 10:00:17,395 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 empty. 2023-02-11 10:00:17,395 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,395 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc empty. 2023-02-11 10:00:17,396 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 empty. 2023-02-11 10:00:17,396 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da empty. 2023-02-11 10:00:17,396 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c empty. 2023-02-11 10:00:17,400 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:17,400 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,400 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:17,401 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da empty. 2023-02-11 10:00:17,401 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,401 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,401 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,401 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676109617344 regions 2023-02-11 10:00:17,426 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:17,427 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(7671): creating {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,428 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(7671): creating {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,428 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(7671): creating {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,470 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,471 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:17,471 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:17,471 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:17,471 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:17,471 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:17,471 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:17,472 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109617471Disabling compacts and flushes for region at 1676109617471Disabling writes for close at 1676109617471Writing region close event to WAL at 1676109617471Closed at 1676109617471 2023-02-11 10:00:17,473 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(7671): creating {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,475 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:17,477 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:17,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:17,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:17,477 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:17,478 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:17,478 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109617477Disabling compacts and flushes for region at 1676109617477Disabling writes for close at 1676109617477Writing region close event to WAL at 1676109617478 (+1 ms)Closed at 1676109617478 2023-02-11 10:00:17,479 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(7671): creating {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,481 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,482 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:17,482 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,482 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,482 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:17,482 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,482 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,482 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109617482Disabling compacts and flushes for region at 1676109617482Disabling writes for close at 1676109617482Writing region close event to WAL at 1676109617482Closed at 1676109617482 2023-02-11 10:00:17,483 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(7671): creating {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:17,502 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,504 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:17,504 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:17,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,505 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,505 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109617504Disabling compacts and flushes for region at 1676109617504Disabling writes for close at 1676109617505 (+1 ms)Writing region close event to WAL at 1676109617505Closed at 1676109617505 2023-02-11 10:00:17,515 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,515 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:17,515 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:17,515 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:17,515 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:17,515 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:17,516 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-2] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109617515Disabling compacts and flushes for region at 1676109617515Disabling writes for close at 1676109617515Writing region close event to WAL at 1676109617516 (+1 ms)Closed at 1676109617516 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:17,516 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:17,516 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:17,516 INFO [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:17,517 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676109617344-pool-1] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109617516Disabling compacts and flushes for region at 1676109617516Disabling writes for close at 1676109617516Writing region close event to WAL at 1676109617516Closed at 1676109617516 2023-02-11 10:00:17,522 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 10:00:17,524 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,524 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,524 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,525 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,525 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,525 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109617523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109617523"}]},"ts":"1676109617523"} 2023-02-11 10:00:17,577 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:17,579 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 10:00:17,579 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109617579"}]},"ts":"1676109617579"} 2023-02-11 10:00:17,582 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:17,586 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:17,590 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:17,590 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:17,590 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:17,591 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:17,591 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:17,594 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:17,594 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:17,595 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:17,595 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:17,596 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:17,597 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:17,597 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:17,597 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:17,598 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:17,599 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:17,600 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:17,600 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:17,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 10:00:17,747 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:17,750 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:17,750 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:17,750 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:17,750 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:17,750 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109617750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617750"}]},"ts":"1676109617750"} 2023-02-11 10:00:17,750 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:17,750 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109617750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617750"}]},"ts":"1676109617750"} 2023-02-11 10:00:17,750 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617750"}]},"ts":"1676109617750"} 2023-02-11 10:00:17,750 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617750"}]},"ts":"1676109617750"} 2023-02-11 10:00:17,750 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617750"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617750"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617750"}]},"ts":"1676109617750"} 2023-02-11 10:00:17,753 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:17,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:17,757 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=11, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:17,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:17,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=14, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:17,769 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:17,769 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617769"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109617769"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109617769"}]},"ts":"1676109617769"} 2023-02-11 10:00:17,773 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=13, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:17,906 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:17,906 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 10:00:17,910 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41800, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 10:00:17,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,917 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:17,917 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 10:00:17,921 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55964, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 10:00:17,921 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,923 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:17,924 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:17,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:17,926 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,927 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:17,928 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:17,928 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:17,928 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,929 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:17,933 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:17,933 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:17,933 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:17,934 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:17,935 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:17,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,937 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:17,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:17,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:17,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:17,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:17,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71588246, jitterRate=0.0667479932308197}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:17,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109617916Writing region info on filesystem at 1676109617916Initializing all the Stores at 1676109617918 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109617918Cleaning up temporary data from old regions at 1676109617938 (+20 ms)Cleaning up detritus from prior splits at 1676109617939 (+1 ms)Running coprocessor post-open hooks at 1676109617956 (+17 ms)Region opened successfully at 1676109617956 2023-02-11 10:00:17,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:17,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69593075, jitterRate=0.03701762855052948}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:17,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109617924Writing region info on filesystem at 1676109617924Initializing all the Stores at 1676109617925 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109617925Cleaning up temporary data from old regions at 1676109617940 (+15 ms)Cleaning up detritus from prior splits at 1676109617941 (+1 ms)Running coprocessor post-open hooks at 1676109617960 (+19 ms)Region opened successfully at 1676109617960 2023-02-11 10:00:17,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=18, masterSystemTime=1676109617916 2023-02-11 10:00:17,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:17,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:17,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:17,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:17,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,967 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:17,968 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617967"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109617967"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109617967"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109617967"}]},"ts":"1676109617967"} 2023-02-11 10:00:17,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=21, masterSystemTime=1676109617906 2023-02-11 10:00:17,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61320896, jitterRate=-0.08624744415283203}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:17,973 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109617926Writing region info on filesystem at 1676109617926Initializing all the Stores at 1676109617927 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109617927Cleaning up temporary data from old regions at 1676109617941 (+14 ms)Cleaning up detritus from prior splits at 1676109617942 (+1 ms)Running coprocessor post-open hooks at 1676109617973 (+31 ms)Region opened successfully at 1676109617974 (+1 ms) 2023-02-11 10:00:17,985 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:17,985 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:17,988 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:17,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=20, masterSystemTime=1676109617917 2023-02-11 10:00:17,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:17,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:17,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:17,993 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:17,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:17,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:17,993 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109617993"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109617993"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109617993"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109617993"}]},"ts":"1676109617993"} 2023-02-11 10:00:17,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:17,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:17,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:17,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=11 2023-02-11 10:00:17,995 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=11, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 214 msec 2023-02-11 10:00:17,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:17,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,000 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:18,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:18,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,002 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:18,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 403 msec 2023-02-11 10:00:18,002 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618002"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109618002"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109618002"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109618002"}]},"ts":"1676109618002"} 2023-02-11 10:00:18,006 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:18,006 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:18,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=13 2023-02-11 10:00:18,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=13, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 228 msec 2023-02-11 10:00:18,007 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:18,008 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:18,011 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:18,011 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:18,012 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:18,012 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:18,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,013 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 415 msec 2023-02-11 10:00:18,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:18,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,015 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=14 2023-02-11 10:00:18,015 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=14, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 245 msec 2023-02-11 10:00:18,016 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:18,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67466569, jitterRate=0.005330219864845276}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:18,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109617967Writing region info on filesystem at 1676109617967Initializing all the Stores at 1676109617969 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109617969Cleaning up temporary data from old regions at 1676109618001 (+32 ms)Cleaning up detritus from prior splits at 1676109618002 (+1 ms)Running coprocessor post-open hooks at 1676109618016 (+14 ms)Region opened successfully at 1676109618016 2023-02-11 10:00:18,018 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 424 msec 2023-02-11 10:00:18,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=17, masterSystemTime=1676109617916 2023-02-11 10:00:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,023 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:18,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618023"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109618023"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109618023"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109618023"}]},"ts":"1676109618023"} 2023-02-11 10:00:18,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:18,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-02-11 10:00:18,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 272 msec 2023-02-11 10:00:18,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:18,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 450 msec 2023-02-11 10:00:18,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74206161, jitterRate=0.10575796663761139}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:18,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60732974, jitterRate=-0.09500816464424133}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:18,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109618002Writing region info on filesystem at 1676109618002Initializing all the Stores at 1676109618004 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109618004Cleaning up temporary data from old regions at 1676109618020 (+16 ms)Cleaning up detritus from prior splits at 1676109618022 (+2 ms)Running coprocessor post-open hooks at 1676109618045 (+23 ms)Region opened successfully at 1676109618045 2023-02-11 10:00:18,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109617995Writing region info on filesystem at 1676109617995Initializing all the Stores at 1676109617997 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109617997Cleaning up temporary data from old regions at 1676109618016 (+19 ms)Cleaning up detritus from prior splits at 1676109618018 (+2 ms)Running coprocessor post-open hooks at 1676109618045 (+27 ms)Region opened successfully at 1676109618045 2023-02-11 10:00:18,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=19, masterSystemTime=1676109617917 2023-02-11 10:00:18,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=16, masterSystemTime=1676109617906 2023-02-11 10:00:18,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,052 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:18,052 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618051"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109618051"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109618051"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109618051"}]},"ts":"1676109618051"} 2023-02-11 10:00:18,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,053 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:18,056 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618053"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109618053"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109618053"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109618053"}]},"ts":"1676109618053"} 2023-02-11 10:00:18,064 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-02-11 10:00:18,064 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 300 msec 2023-02-11 10:00:18,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-02-11 10:00:18,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 308 msec 2023-02-11 10:00:18,067 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 473 msec 2023-02-11 10:00:18,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-11 10:00:18,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 474 msec 2023-02-11 10:00:18,070 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 10:00:18,070 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109618070"}]},"ts":"1676109618070"} 2023-02-11 10:00:18,073 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:18,076 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676109617344 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 10:00:18,079 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676109617344 in 699 msec 2023-02-11 10:00:18,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-11 10:00:18,155 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 9 completed 2023-02-11 10:00:18,162 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676109617344 2023-02-11 10:00:18,163 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,166 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:18,185 DEBUG [hconnection-0x6de71207-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:18,188 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41802, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:18,192 DEBUG [hconnection-0x6de71207-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:18,195 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55972, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:18,205 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:18,208 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:18,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:18,219 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109618219"}]},"ts":"1676109618219"} 2023-02-11 10:00:18,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 10:00:18,221 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:18,224 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:18,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:18,229 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:18,229 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:18,229 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:18,229 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:18,230 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:18,231 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:18,231 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618231"}]},"ts":"1676109618231"} 2023-02-11 10:00:18,232 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:18,232 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618232"}]},"ts":"1676109618232"} 2023-02-11 10:00:18,234 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:18,234 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:18,234 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618234"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618234"}]},"ts":"1676109618234"} 2023-02-11 10:00:18,234 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618234"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618234"}]},"ts":"1676109618234"} 2023-02-11 10:00:18,234 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:18,235 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618234"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618234"}]},"ts":"1676109618234"} 2023-02-11 10:00:18,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=24, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:18,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:18,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:18,240 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:18,243 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:18,250 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:18,252 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:18,253 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109618252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109618252"}]},"ts":"1676109618252"} 2023-02-11 10:00:18,255 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:18,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:18,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:18,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:18,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:18,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:18,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:18,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:18,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:18,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:18,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:18,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:18,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:18,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:18,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:18,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:18,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:18,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109618407Running coprocessor pre-close hooks at 1676109618407Disabling compacts and flushes for region at 1676109618407Disabling writes for close at 1676109618409 (+2 ms)Writing region close event to WAL at 1676109618409Running coprocessor post-close hooks at 1676109618421 (+12 ms)Closed at 1676109618421 2023-02-11 10:00:18,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109618403Running coprocessor pre-close hooks at 1676109618404 (+1 ms)Disabling compacts and flushes for region at 1676109618404Disabling writes for close at 1676109618409 (+5 ms)Writing region close event to WAL at 1676109618412 (+3 ms)Running coprocessor post-close hooks at 1676109618421 (+9 ms)Closed at 1676109618421 2023-02-11 10:00:18,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:18,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109618408Running coprocessor pre-close hooks at 1676109618408Disabling compacts and flushes for region at 1676109618408Disabling writes for close at 1676109618412 (+4 ms)Writing region close event to WAL at 1676109618414 (+2 ms)Running coprocessor post-close hooks at 1676109618423 (+9 ms)Closed at 1676109618424 (+1 ms) 2023-02-11 10:00:18,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:18,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:18,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:18,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 1 ms 2023-02-11 10:00:18,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,426 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:18,426 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618426"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618426"}]},"ts":"1676109618426"} 2023-02-11 10:00:18,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:18,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:18,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:18,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,435 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:18,435 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618435"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618435"}]},"ts":"1676109618435"} 2023-02-11 10:00:18,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:18,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:18,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:18,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:18,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,437 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:18,437 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618437"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618437"}]},"ts":"1676109618437"} 2023-02-11 10:00:18,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=24 2023-02-11 10:00:18,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=24, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 194 msec 2023-02-11 10:00:18,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:18,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109618424Running coprocessor pre-close hooks at 1676109618424Disabling compacts and flushes for region at 1676109618424Disabling writes for close at 1676109618426 (+2 ms)Writing region close event to WAL at 1676109618428 (+2 ms)Running coprocessor post-close hooks at 1676109618440 (+12 ms)Closed at 1676109618440 2023-02-11 10:00:18,444 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 213 msec 2023-02-11 10:00:18,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:18,446 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-11 10:00:18,446 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 198 msec 2023-02-11 10:00:18,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,447 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:18,447 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618447"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618447"}]},"ts":"1676109618447"} 2023-02-11 10:00:18,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-11 10:00:18,448 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 202 msec 2023-02-11 10:00:18,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:18,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:18,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109618431Running coprocessor pre-close hooks at 1676109618431Disabling compacts and flushes for region at 1676109618431Disabling writes for close at 1676109618431Writing region close event to WAL at 1676109618432 (+1 ms)Running coprocessor post-close hooks at 1676109618450 (+18 ms)Closed at 1676109618450 2023-02-11 10:00:18,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 220 msec 2023-02-11 10:00:18,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:18,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109618436Running coprocessor pre-close hooks at 1676109618436Disabling compacts and flushes for region at 1676109618436Disabling writes for close at 1676109618437 (+1 ms)Writing region close event to WAL at 1676109618437Running coprocessor post-close hooks at 1676109618451 (+14 ms)Closed at 1676109618451 2023-02-11 10:00:18,452 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 222 msec 2023-02-11 10:00:18,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:18,454 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:18,454 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-11 10:00:18,454 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109618454"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618454"}]},"ts":"1676109618454"} 2023-02-11 10:00:18,454 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 207 msec 2023-02-11 10:00:18,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:18,455 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:18,455 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109618455"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109618455"}]},"ts":"1676109618455"} 2023-02-11 10:00:18,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 228 msec 2023-02-11 10:00:18,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-02-11 10:00:18,460 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 219 msec 2023-02-11 10:00:18,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-11 10:00:18,461 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 203 msec 2023-02-11 10:00:18,462 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 234 msec 2023-02-11 10:00:18,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=22 2023-02-11 10:00:18,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 235 msec 2023-02-11 10:00:18,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109618465"}]},"ts":"1676109618465"} 2023-02-11 10:00:18,467 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:18,470 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:18,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 10:00:18,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 261 msec 2023-02-11 10:00:18,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-11 10:00:18,974 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 22 completed 2023-02-11 10:00:18,985 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109618986 (current time:1676109618986). 2023-02-11 10:00:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x1eb52df7 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:18,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43c566cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:19,000 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33810, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:19,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eb52df7 to 127.0.0.1:64863 2023-02-11 10:00:19,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:19,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:19,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:19,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:19,027 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:19,028 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110219028 2023-02-11 10:00:19,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:19,029 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:19,033 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:19,033 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:19,052 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:19,056 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,056 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,056 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,056 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,056 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,057 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,057 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,062 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:19,073 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,073 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,073 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,073 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,074 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,074 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,075 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:19,153 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:19,154 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676109617344 2023-02-11 10:00:19,156 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109617344 2023-02-11 10:00:19,204 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109617344 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/emptySnaptb-1676109617344 2023-02-11 10:00:19,239 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676109617344 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:19,239 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109617344 2023-02-11 10:00:19,239 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109617344 2023-02-11 10:00:19,240 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109619029Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109619052 (+23 ms)Consolidate snapshot: emptySnaptb-1676109617344 at 1676109619154 (+102 ms)Loading Region manifests for emptySnaptb-1676109617344 at 1676109619154Writing data manifest for emptySnaptb-1676109617344 at 1676109619169 (+15 ms)Verifying snapshot: emptySnaptb-1676109617344 at 1676109619189 (+20 ms)Snapshot emptySnaptb-1676109617344 of table testRestoreSnapshot-1676109617344 completed at 1676109619239 (+50 ms) 2023-02-11 10:00:19,241 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:19,243 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 223 msec 2023-02-11 10:00:19,283 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:19,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:19,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:19,289 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:19,293 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676109617344 2023-02-11 10:00:19,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:19,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 10:00:19,305 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676109617344 2023-02-11 10:00:19,305 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109619305"}]},"ts":"1676109619305"} 2023-02-11 10:00:19,307 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:19,312 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:19,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:19,316 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:19,316 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:19,316 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:19,317 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:19,318 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,318 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,318 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:19,318 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:19,318 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,318 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:19,318 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:19,318 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,319 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:19,320 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:19,321 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,321 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:19,322 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,322 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:19,468 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:19,469 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,469 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,469 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,469 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619469"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619469"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619469"}]},"ts":"1676109619469"} 2023-02-11 10:00:19,469 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109619469"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619469"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619469"}]},"ts":"1676109619469"} 2023-02-11 10:00:19,469 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,469 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,470 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619469"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619469"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619469"}]},"ts":"1676109619469"} 2023-02-11 10:00:19,470 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619469"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619469"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619469"}]},"ts":"1676109619469"} 2023-02-11 10:00:19,469 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109619469"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619469"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619469"}]},"ts":"1676109619469"} 2023-02-11 10:00:19,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:19,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=41, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:19,477 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=38, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:19,479 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=39, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:19,480 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,481 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619480"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109619480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109619480"}]},"ts":"1676109619480"} 2023-02-11 10:00:19,482 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=37, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:19,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=40, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:19,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 10:00:19,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:19,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:19,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:19,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:19,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,634 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,635 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,636 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:19,636 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:19,636 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:19,638 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:19,639 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:19,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:19,640 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:19,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,641 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:19,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,643 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,644 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,646 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:19,646 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:19,646 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:19,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,648 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:19,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74926453, jitterRate=0.1164911538362503}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109619633Writing region info on filesystem at 1676109619633Initializing all the Stores at 1676109619634 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619634Cleaning up temporary data from old regions at 1676109619645 (+11 ms)Cleaning up detritus from prior splits at 1676109619645Running coprocessor post-open hooks at 1676109619650 (+5 ms)Region opened successfully at 1676109619650 2023-02-11 10:00:19,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:19,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=43, masterSystemTime=1676109619626 2023-02-11 10:00:19,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:19,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:19,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:19,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:19,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,654 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,654 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,655 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109619654"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619654"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619654"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619654"}]},"ts":"1676109619654"} 2023-02-11 10:00:19,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69066708, jitterRate=0.02917414903640747}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109619634Writing region info on filesystem at 1676109619634Initializing all the Stores at 1676109619635 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619635Cleaning up temporary data from old regions at 1676109619647 (+12 ms)Cleaning up detritus from prior splits at 1676109619648 (+1 ms)Running coprocessor post-open hooks at 1676109619655 (+7 ms)Region opened successfully at 1676109619655 2023-02-11 10:00:19,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=44, masterSystemTime=1676109619628 2023-02-11 10:00:19,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:19,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:19,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:19,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:19,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,661 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:19,661 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619661"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619661"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619661"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619661"}]},"ts":"1676109619661"} 2023-02-11 10:00:19,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73931931, jitterRate=0.10167162120342255}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-11 10:00:19,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109619640Writing region info on filesystem at 1676109619640Initializing all the Stores at 1676109619642 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619642Cleaning up temporary data from old regions at 1676109619652 (+10 ms)Cleaning up detritus from prior splits at 1676109619656 (+4 ms)Running coprocessor post-open hooks at 1676109619662 (+6 ms)Region opened successfully at 1676109619662 2023-02-11 10:00:19,663 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 186 msec 2023-02-11 10:00:19,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=48, masterSystemTime=1676109619635 2023-02-11 10:00:19,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 351 msec 2023-02-11 10:00:19,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:19,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:19,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:19,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:19,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:19,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,668 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,668 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,669 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619668"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619668"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619668"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619668"}]},"ts":"1676109619668"} 2023-02-11 10:00:19,669 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,671 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=41 2023-02-11 10:00:19,670 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:19,672 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=41, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 190 msec 2023-02-11 10:00:19,672 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:19,673 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:19,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 359 msec 2023-02-11 10:00:19,674 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,676 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:19,676 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:19,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=40 2023-02-11 10:00:19,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=40, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 187 msec 2023-02-11 10:00:19,676 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:19,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,678 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 364 msec 2023-02-11 10:00:19,680 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:19,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62771742, jitterRate=-0.06462815403938293}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109619661Writing region info on filesystem at 1676109619661Initializing all the Stores at 1676109619663 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619664 (+1 ms)Cleaning up temporary data from old regions at 1676109619679 (+15 ms)Cleaning up detritus from prior splits at 1676109619679Running coprocessor post-open hooks at 1676109619684 (+5 ms)Region opened successfully at 1676109619684 2023-02-11 10:00:19,684 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=46, masterSystemTime=1676109619628 2023-02-11 10:00:19,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:19,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:19,688 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:19,688 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619688"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619688"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619688"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619688"}]},"ts":"1676109619688"} 2023-02-11 10:00:19,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:19,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61485220, jitterRate=-0.08379882574081421}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109619655Writing region info on filesystem at 1676109619655Initializing all the Stores at 1676109619657 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619658 (+1 ms)Cleaning up temporary data from old regions at 1676109619684 (+26 ms)Cleaning up detritus from prior splits at 1676109619685 (+1 ms)Running coprocessor post-open hooks at 1676109619691 (+6 ms)Region opened successfully at 1676109619691 2023-02-11 10:00:19,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=45, masterSystemTime=1676109619626 2023-02-11 10:00:19,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:19,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:19,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=39 2023-02-11 10:00:19,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=39, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 212 msec 2023-02-11 10:00:19,695 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:19,695 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109619695"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619695"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619695"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619695"}]},"ts":"1676109619695"} 2023-02-11 10:00:19,696 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:19,696 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:19,697 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:19,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 383 msec 2023-02-11 10:00:19,699 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:19,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=38 2023-02-11 10:00:19,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=38, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 221 msec 2023-02-11 10:00:19,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,703 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 389 msec 2023-02-11 10:00:19,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:19,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59889451, jitterRate=-0.10757763683795929}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:19,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109619668Writing region info on filesystem at 1676109619668Initializing all the Stores at 1676109619670 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109619670Cleaning up temporary data from old regions at 1676109619704 (+34 ms)Cleaning up detritus from prior splits at 1676109619704Running coprocessor post-open hooks at 1676109619709 (+5 ms)Region opened successfully at 1676109619709 2023-02-11 10:00:19,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=47, masterSystemTime=1676109619635 2023-02-11 10:00:19,712 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:19,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:19,713 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:19,713 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109619713"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109619713"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109619713"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109619713"}]},"ts":"1676109619713"} 2023-02-11 10:00:19,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=37 2023-02-11 10:00:19,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=37, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 233 msec 2023-02-11 10:00:19,720 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-11 10:00:19,720 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 405 msec 2023-02-11 10:00:19,721 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109619721"}]},"ts":"1676109619721"} 2023-02-11 10:00:19,723 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:19,727 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676109617344' was successfully enabled. 2023-02-11 10:00:19,731 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676109617344 in 434 msec 2023-02-11 10:00:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-11 10:00:20,058 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 36 completed 2023-02-11 10:00:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:20,127 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676109617344 2023-02-11 10:00:20,127 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:20,129 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:20,157 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:20,161 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:20,163 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:20,166 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:20,168 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:20,179 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:20,179 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:20,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:20,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 10:00:20,185 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109620185"}]},"ts":"1676109620185"} 2023-02-11 10:00:20,187 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:20,189 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:20,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:20,197 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:20,198 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:20,198 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:20,198 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:20,199 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:20,200 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:20,201 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:20,201 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620200"}]},"ts":"1676109620200"} 2023-02-11 10:00:20,201 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109620200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620200"}]},"ts":"1676109620200"} 2023-02-11 10:00:20,201 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:20,202 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109620201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620201"}]},"ts":"1676109620201"} 2023-02-11 10:00:20,202 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:20,202 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620202"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620202"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620202"}]},"ts":"1676109620202"} 2023-02-11 10:00:20,202 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:20,202 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620202"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620202"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620202"}]},"ts":"1676109620202"} 2023-02-11 10:00:20,205 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=51, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:20,207 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=50, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:20,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=55, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:20,213 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=53, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:20,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=54, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:20,220 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:20,222 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:20,222 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620222"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109620222"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109620222"}]},"ts":"1676109620222"} 2023-02-11 10:00:20,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=52, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:20,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:20,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:20,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:20,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:20,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:20,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:20,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b714c90af14d8acfa1b06ca5768827da 1/1 column families, dataSize=22.57 KB heapSize=48.77 KB 2023-02-11 10:00:20,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:20,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:20,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:20,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:20,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:20,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:20,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing eb9c4974f7474c0e09c82c0ecda0c585 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 10:00:20,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:20,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:20,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:20,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:20,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:20,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:20,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 59ce432d7225c2fa47a6ee02f27bbfdc 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 10:00:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 10:00:20,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:20,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/.tmp/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:20,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/.tmp/cf/c8e02b793157422b98df2d998ef0b4c8 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:20,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8, entries=27, sequenceid=8, filesize=9.5 K 2023-02-11 10:00:20,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for eb9c4974f7474c0e09c82c0ecda0c585 in 184ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:20,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676109617344' 2023-02-11 10:00:20,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:20,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:20,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109620364Running coprocessor pre-close hooks at 1676109620364Disabling compacts and flushes for region at 1676109620364Disabling writes for close at 1676109620364Obtaining lock to block concurrent updates at 1676109620364Preparing flush snapshotting stores in eb9c4974f7474c0e09c82c0ecda0c585 at 1676109620364Finished memstore snapshotting testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676109620371 (+7 ms)Flushing stores of testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. at 1676109620372 (+1 ms)Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: creating writer at 1676109620376 (+4 ms)Flushing mob file eb9c4974f7474c0e09c82c0ecda0c585/cf: appending metadata at 1676109620417 (+41 ms)Flushing mob file eb9c4974f7474c0e09c82c0ecda0c585/cf: closing flushed file at 1676109620417Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: appending metadata at 1676109620501 (+84 ms)Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: closing flushed file at 1676109620506 (+5 ms)Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: reopening flushed file at 1676109620538 (+32 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for eb9c4974f7474c0e09c82c0ecda0c585 in 184ms, sequenceid=8, compaction requested=false at 1676109620548 (+10 ms)Writing region close event to WAL at 1676109620555 (+7 ms)Running coprocessor post-close hooks at 1676109620559 (+4 ms)Closed at 1676109620559 2023-02-11 10:00:20,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:20,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:20,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:20,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:20,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:20,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:20,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:20,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1bff2e862183c9fd81aa3c886062178c 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-11 10:00:20,562 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:20,562 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109620562"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109620562"}]},"ts":"1676109620562"} 2023-02-11 10:00:20,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=50 2023-02-11 10:00:20,568 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=50, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 358 msec 2023-02-11 10:00:20,569 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 377 msec 2023-02-11 10:00:20,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:20,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/.tmp/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:20,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/.tmp/cf/f700cba99b9d45a6939369b56232ede4 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:20,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4, entries=40, sequenceid=8, filesize=11.7 K 2023-02-11 10:00:20,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 1bff2e862183c9fd81aa3c886062178c in 78ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:20,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:20,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:20,651 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109620562Running coprocessor pre-close hooks at 1676109620562Disabling compacts and flushes for region at 1676109620562Disabling writes for close at 1676109620562Obtaining lock to block concurrent updates at 1676109620562Preparing flush snapshotting stores in 1bff2e862183c9fd81aa3c886062178c at 1676109620562Finished memstore snapshotting testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., syncing WAL and waiting on mvcc, flushsize=dataSize=2678, getHeapSize=6000, getOffHeapSize=0, getCellsCount=40 at 1676109620562Flushing stores of testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. at 1676109620563 (+1 ms)Flushing 1bff2e862183c9fd81aa3c886062178c/cf: creating writer at 1676109620563Flushing mob file 1bff2e862183c9fd81aa3c886062178c/cf: appending metadata at 1676109620570 (+7 ms)Flushing mob file 1bff2e862183c9fd81aa3c886062178c/cf: closing flushed file at 1676109620570Flushing 1bff2e862183c9fd81aa3c886062178c/cf: appending metadata at 1676109620592 (+22 ms)Flushing 1bff2e862183c9fd81aa3c886062178c/cf: closing flushed file at 1676109620592Flushing 1bff2e862183c9fd81aa3c886062178c/cf: reopening flushed file at 1676109620631 (+39 ms)Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 1bff2e862183c9fd81aa3c886062178c in 78ms, sequenceid=8, compaction requested=false at 1676109620640 (+9 ms)Writing region close event to WAL at 1676109620646 (+6 ms)Running coprocessor post-close hooks at 1676109620651 (+5 ms)Closed at 1676109620651 2023-02-11 10:00:20,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:20,657 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:20,658 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620657"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109620657"}]},"ts":"1676109620657"} 2023-02-11 10:00:20,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=53 2023-02-11 10:00:20,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=53, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 447 msec 2023-02-11 10:00:20,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 471 msec 2023-02-11 10:00:20,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:20,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:20,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/.tmp/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:20,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/.tmp/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:20,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/.tmp/cf/fa1f306b33c8410e8fc7746a0d41671f as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:20,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/.tmp/cf/3719691d68774d919d12001277aeabf7 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:20,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7, entries=345, sequenceid=8, filesize=63.7 K 2023-02-11 10:00:20,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f, entries=27, sequenceid=8, filesize=9.5 K 2023-02-11 10:00:20,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 59ce432d7225c2fa47a6ee02f27bbfdc in 527ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:20,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for b714c90af14d8acfa1b06ca5768827da in 541ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:20,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:20,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:20,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:20,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109620374Running coprocessor pre-close hooks at 1676109620374Disabling compacts and flushes for region at 1676109620374Disabling writes for close at 1676109620375 (+1 ms)Obtaining lock to block concurrent updates at 1676109620375Preparing flush snapshotting stores in 59ce432d7225c2fa47a6ee02f27bbfdc at 1676109620375Finished memstore snapshotting testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676109620375Flushing stores of testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. at 1676109620375Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: creating writer at 1676109620376 (+1 ms)Flushing mob file 59ce432d7225c2fa47a6ee02f27bbfdc/cf: appending metadata at 1676109620417 (+41 ms)Flushing mob file 59ce432d7225c2fa47a6ee02f27bbfdc/cf: closing flushed file at 1676109620417Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: appending metadata at 1676109620866 (+449 ms)Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: closing flushed file at 1676109620866Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: reopening flushed file at 1676109620892 (+26 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 59ce432d7225c2fa47a6ee02f27bbfdc in 527ms, sequenceid=8, compaction requested=false at 1676109620902 (+10 ms)Writing region close event to WAL at 1676109620907 (+5 ms)Running coprocessor post-close hooks at 1676109620912 (+5 ms)Closed at 1676109620912 2023-02-11 10:00:20,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:20,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109620360Running coprocessor pre-close hooks at 1676109620360Disabling compacts and flushes for region at 1676109620360Disabling writes for close at 1676109620361 (+1 ms)Obtaining lock to block concurrent updates at 1676109620362 (+1 ms)Preparing flush snapshotting stores in b714c90af14d8acfa1b06ca5768827da at 1676109620362Finished memstore snapshotting testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., syncing WAL and waiting on mvcc, flushsize=dataSize=23111, getHeapSize=49920, getOffHeapSize=0, getCellsCount=345 at 1676109620371 (+9 ms)Flushing stores of testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. at 1676109620372 (+1 ms)Flushing b714c90af14d8acfa1b06ca5768827da/cf: creating writer at 1676109620376 (+4 ms)Flushing mob file b714c90af14d8acfa1b06ca5768827da/cf: appending metadata at 1676109620431 (+55 ms)Flushing mob file b714c90af14d8acfa1b06ca5768827da/cf: closing flushed file at 1676109620431Flushing b714c90af14d8acfa1b06ca5768827da/cf: appending metadata at 1676109620866 (+435 ms)Flushing b714c90af14d8acfa1b06ca5768827da/cf: closing flushed file at 1676109620866Flushing b714c90af14d8acfa1b06ca5768827da/cf: reopening flushed file at 1676109620893 (+27 ms)Finished flush of dataSize ~22.57 KB/23111, heapSize ~48.75 KB/49920, currentSize=0 B/0 for b714c90af14d8acfa1b06ca5768827da in 541ms, sequenceid=8, compaction requested=false at 1676109620903 (+10 ms)Writing region close event to WAL at 1676109620907 (+4 ms)Running coprocessor post-close hooks at 1676109620912 (+5 ms)Closed at 1676109620912 2023-02-11 10:00:20,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:20,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:20,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:20,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:20,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:20,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:20,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:20,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 86a1312865cab1cc55298ab2c74451da 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-11 10:00:20,916 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:20,917 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109620916"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109620916"}]},"ts":"1676109620916"} 2023-02-11 10:00:20,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:20,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:20,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:20,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:20,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:20,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:20,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:20,918 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:20,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4f728331505b601d10a557ad43850320 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-11 10:00:20,918 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109620918"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109620918"}]},"ts":"1676109620918"} 2023-02-11 10:00:20,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=52 2023-02-11 10:00:20,925 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=52, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 694 msec 2023-02-11 10:00:20,928 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=55 2023-02-11 10:00:20,928 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=55, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 712 msec 2023-02-11 10:00:20,930 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 734 msec 2023-02-11 10:00:20,931 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 737 msec 2023-02-11 10:00:20,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 10:00:20,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:20,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:20,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/.tmp/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:20,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/.tmp/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:20,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/.tmp/cf/ca87f195a3104b539048984ba5a3a37c as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:20,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/.tmp/cf/7558e76cf7794bd9821d73896593011f as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:20,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c, entries=27, sequenceid=8, filesize=9.5 K 2023-02-11 10:00:20,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 86a1312865cab1cc55298ab2c74451da in 77ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:20,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f, entries=34, sequenceid=8, filesize=10.7 K 2023-02-11 10:00:20,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 4f728331505b601d10a557ad43850320 in 81ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:21,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:21,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:21,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:21,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109620915Running coprocessor pre-close hooks at 1676109620915Disabling compacts and flushes for region at 1676109620915Disabling writes for close at 1676109620916 (+1 ms)Obtaining lock to block concurrent updates at 1676109620916Preparing flush snapshotting stores in 86a1312865cab1cc55298ab2c74451da at 1676109620916Finished memstore snapshotting testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676109620916Flushing stores of testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. at 1676109620917 (+1 ms)Flushing 86a1312865cab1cc55298ab2c74451da/cf: creating writer at 1676109620917Flushing mob file 86a1312865cab1cc55298ab2c74451da/cf: appending metadata at 1676109620928 (+11 ms)Flushing mob file 86a1312865cab1cc55298ab2c74451da/cf: closing flushed file at 1676109620928Flushing 86a1312865cab1cc55298ab2c74451da/cf: appending metadata at 1676109620959 (+31 ms)Flushing 86a1312865cab1cc55298ab2c74451da/cf: closing flushed file at 1676109620959Flushing 86a1312865cab1cc55298ab2c74451da/cf: reopening flushed file at 1676109620982 (+23 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 86a1312865cab1cc55298ab2c74451da in 77ms, sequenceid=8, compaction requested=false at 1676109620993 (+11 ms)Writing region close event to WAL at 1676109620999 (+6 ms)Running coprocessor post-close hooks at 1676109621006 (+7 ms)Closed at 1676109621006 2023-02-11 10:00:21,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:21,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109620917Running coprocessor pre-close hooks at 1676109620917Disabling compacts and flushes for region at 1676109620917Disabling writes for close at 1676109620918 (+1 ms)Obtaining lock to block concurrent updates at 1676109620918Preparing flush snapshotting stores in 4f728331505b601d10a557ad43850320 at 1676109620918Finished memstore snapshotting testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676109620919 (+1 ms)Flushing stores of testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. at 1676109620919Flushing 4f728331505b601d10a557ad43850320/cf: creating writer at 1676109620920 (+1 ms)Flushing mob file 4f728331505b601d10a557ad43850320/cf: appending metadata at 1676109620931 (+11 ms)Flushing mob file 4f728331505b601d10a557ad43850320/cf: closing flushed file at 1676109620931Flushing 4f728331505b601d10a557ad43850320/cf: appending metadata at 1676109620962 (+31 ms)Flushing 4f728331505b601d10a557ad43850320/cf: closing flushed file at 1676109620962Flushing 4f728331505b601d10a557ad43850320/cf: reopening flushed file at 1676109620988 (+26 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 4f728331505b601d10a557ad43850320 in 81ms, sequenceid=8, compaction requested=false at 1676109620999 (+11 ms)Writing region close event to WAL at 1676109621002 (+3 ms)Running coprocessor post-close hooks at 1676109621009 (+7 ms)Closed at 1676109621009 2023-02-11 10:00:21,009 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:21,010 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:21,010 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109621010"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109621010"}]},"ts":"1676109621010"} 2023-02-11 10:00:21,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:21,012 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:21,013 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109621012"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109621012"}]},"ts":"1676109621012"} 2023-02-11 10:00:21,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=54 2023-02-11 10:00:21,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=54, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 799 msec 2023-02-11 10:00:21,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=51 2023-02-11 10:00:21,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 824 msec 2023-02-11 10:00:21,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=51, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 809 msec 2023-02-11 10:00:21,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=49 2023-02-11 10:00:21,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 826 msec 2023-02-11 10:00:21,020 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109621020"}]},"ts":"1676109621020"} 2023-02-11 10:00:21,022 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:21,024 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:21,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 846 msec 2023-02-11 10:00:21,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-11 10:00:21,690 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 49 completed 2023-02-11 10:00:21,691 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109621691 (current time:1676109621691). 2023-02-11 10:00:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x6bb7d951 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e7c3abc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:21,706 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33820, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bb7d951 to 127.0.0.1:64863 2023-02-11 10:00:21,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:21,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:21,713 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:21,713 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110221713 2023-02-11 10:00:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:21,713 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:21,714 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:21,714 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:21,732 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:21,741 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,741 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,741 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,741 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,742 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,742 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,743 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,743 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,744 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8] hfiles 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4] hfiles 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f] hfiles 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7] hfiles 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:21,751 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:21,752 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c] hfiles 2023-02-11 10:00:21,752 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f] hfiles 2023-02-11 10:00:21,752 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:21,752 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa] hfiles 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:21,788 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:21,869 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:21,937 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 10:00:21,938 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 10:00:21,964 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:21,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:21,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' is still in progress! 2023-02-11 10:00:21,966 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-11 10:00:22,224 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:22,225 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676109617344 2023-02-11 10:00:22,226 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109617344 2023-02-11 10:00:22,270 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109617344 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/snaptb0-1676109617344 2023-02-11 10:00:22,298 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676109617344 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:22,298 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109617344 2023-02-11 10:00:22,299 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109617344 2023-02-11 10:00:22,299 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109621714Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109621732 (+18 ms)Consolidate snapshot: snaptb0-1676109617344 at 1676109622225 (+493 ms)Loading Region manifests for snaptb0-1676109617344 at 1676109622225Writing data manifest for snaptb0-1676109617344 at 1676109622237 (+12 ms)Verifying snapshot: snaptb0-1676109617344 at 1676109622253 (+16 ms)Snapshot snaptb0-1676109617344 of table testRestoreSnapshot-1676109617344 completed at 1676109622298 (+45 ms) 2023-02-11 10:00:22,301 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:22,302 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 590 msec 2023-02-11 10:00:22,467 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:22,469 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:22,469 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676109617344 2023-02-11 10:00:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 10:00:22,473 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676109617344 2023-02-11 10:00:22,473 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109622473"}]},"ts":"1676109622473"} 2023-02-11 10:00:22,475 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:22,480 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:22,480 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:22,482 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:22,482 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:22,482 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:22,482 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:22,483 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,483 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:22,483 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,483 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,483 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:22,483 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:22,483 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,483 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:22,483 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:22,484 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:22,484 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,484 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,484 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:22,484 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:22,633 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:22,633 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,633 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,634 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109622633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622633"}]},"ts":"1676109622633"} 2023-02-11 10:00:22,633 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,634 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109622633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622633"}]},"ts":"1676109622633"} 2023-02-11 10:00:22,633 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,634 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622633"}]},"ts":"1676109622633"} 2023-02-11 10:00:22,634 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622633"}]},"ts":"1676109622633"} 2023-02-11 10:00:22,633 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622633"}]},"ts":"1676109622633"} 2023-02-11 10:00:22,635 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=64, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:22,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=69, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:22,637 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=65, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:22,639 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=68, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:22,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=66, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:22,641 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,641 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622641"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109622641"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109622641"}]},"ts":"1676109622641"} 2023-02-11 10:00:22,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=67, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 10:00:22,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:22,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:22,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:22,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:22,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,796 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,797 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,797 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,798 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:22,798 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:22,798 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:22,798 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:22,798 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:22,798 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:22,798 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:22,799 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:22,799 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:22,810 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:22,810 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:22,810 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,811 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,811 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:22,811 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:22,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:22,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:22,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65362903, jitterRate=-0.026016846299171448}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109622795Writing region info on filesystem at 1676109622795Initializing all the Stores at 1676109622796 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622796Cleaning up temporary data from old regions at 1676109622815 (+19 ms)Cleaning up detritus from prior splits at 1676109622815Running coprocessor post-open hooks at 1676109622820 (+5 ms)Region opened successfully at 1676109622820 2023-02-11 10:00:22,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60652328, jitterRate=-0.09620988368988037}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73846318, jitterRate=0.10039588809013367}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109622794Writing region info on filesystem at 1676109622794Initializing all the Stores at 1676109622795 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622795Cleaning up temporary data from old regions at 1676109622815 (+20 ms)Cleaning up detritus from prior splits at 1676109622816 (+1 ms)Running coprocessor post-open hooks at 1676109622820 (+4 ms)Region opened successfully at 1676109622820 2023-02-11 10:00:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109622794Writing region info on filesystem at 1676109622794Initializing all the Stores at 1676109622795 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622796 (+1 ms)Cleaning up temporary data from old regions at 1676109622815 (+19 ms)Cleaning up detritus from prior splits at 1676109622815Running coprocessor post-open hooks at 1676109622820 (+5 ms)Region opened successfully at 1676109622820 2023-02-11 10:00:22,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=74, masterSystemTime=1676109622791 2023-02-11 10:00:22,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=72, masterSystemTime=1676109622790 2023-02-11 10:00:22,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=70, masterSystemTime=1676109622789 2023-02-11 10:00:22,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:22,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:22,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:22,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,824 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,824 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109622824"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622824"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622824"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622824"}]},"ts":"1676109622824"} 2023-02-11 10:00:22,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:22,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:22,826 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,826 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622826"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622826"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622826"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622826"}]},"ts":"1676109622826"} 2023-02-11 10:00:22,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:22,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:22,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:22,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,828 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622828"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622828"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622828"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622828"}]},"ts":"1676109622828"} 2023-02-11 10:00:22,828 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,829 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,831 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,831 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:22,831 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:22,832 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:22,832 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:22,832 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:22,832 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=64 2023-02-11 10:00:22,832 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=64, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 193 msec 2023-02-11 10:00:22,833 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:22,833 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:22,833 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:22,834 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:22,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=66 2023-02-11 10:00:22,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=66, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 190 msec 2023-02-11 10:00:22,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 352 msec 2023-02-11 10:00:22,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=65 2023-02-11 10:00:22,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=65, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 194 msec 2023-02-11 10:00:22,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 354 msec 2023-02-11 10:00:22,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 357 msec 2023-02-11 10:00:22,844 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:22,845 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,848 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:22,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,848 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,851 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:22,851 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:22,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:22,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68357153, jitterRate=0.018600955605506897}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109622827Writing region info on filesystem at 1676109622827Initializing all the Stores at 1676109622828 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622829 (+1 ms)Cleaning up temporary data from old regions at 1676109622848 (+19 ms)Cleaning up detritus from prior splits at 1676109622851 (+3 ms)Running coprocessor post-open hooks at 1676109622855 (+4 ms)Region opened successfully at 1676109622855 2023-02-11 10:00:22,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=73, masterSystemTime=1676109622791 2023-02-11 10:00:22,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:22,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:22,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59742931, jitterRate=-0.10976095497608185}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109622824Writing region info on filesystem at 1676109622824Initializing all the Stores at 1676109622828 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622828Cleaning up temporary data from old regions at 1676109622854 (+26 ms)Cleaning up detritus from prior splits at 1676109622854Running coprocessor post-open hooks at 1676109622859 (+5 ms)Region opened successfully at 1676109622859 2023-02-11 10:00:22,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:22,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:22,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72989995, jitterRate=0.08763568103313446}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:22,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=75, masterSystemTime=1676109622789 2023-02-11 10:00:22,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109622828Writing region info on filesystem at 1676109622828Initializing all the Stores at 1676109622830 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109622830Cleaning up temporary data from old regions at 1676109622855 (+25 ms)Cleaning up detritus from prior splits at 1676109622855Running coprocessor post-open hooks at 1676109622860 (+5 ms)Region opened successfully at 1676109622860 2023-02-11 10:00:22,860 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:22,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=71, masterSystemTime=1676109622790 2023-02-11 10:00:22,861 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622859"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622859"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622859"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622859"}]},"ts":"1676109622859"} 2023-02-11 10:00:22,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:22,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:22,864 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:22,864 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109622864"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622864"}]},"ts":"1676109622864"} 2023-02-11 10:00:22,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:22,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:22,865 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:22,865 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109622865"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109622865"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109622865"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109622865"}]},"ts":"1676109622865"} 2023-02-11 10:00:22,868 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=68 2023-02-11 10:00:22,868 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=68, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 224 msec 2023-02-11 10:00:22,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=67 2023-02-11 10:00:22,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=67, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 223 msec 2023-02-11 10:00:22,869 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 388 msec 2023-02-11 10:00:22,870 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=69 2023-02-11 10:00:22,870 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=69, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 231 msec 2023-02-11 10:00:22,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 389 msec 2023-02-11 10:00:22,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=69, resume processing ppid=63 2023-02-11 10:00:22,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 390 msec 2023-02-11 10:00:22,873 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109622873"}]},"ts":"1676109622873"} 2023-02-11 10:00:22,874 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:22,877 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676109617344' was successfully enabled. 2023-02-11 10:00:22,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676109617344 in 409 msec 2023-02-11 10:00:23,137 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 10:00:23,137 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-11 10:00:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-11 10:00:23,225 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 63 completed 2023-02-11 10:00:23,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:23,258 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676109617344 2023-02-11 10:00:23,258 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:23,260 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:23,493 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:23,683 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:23,864 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:24,073 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:24,210 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:25,852 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x14908088 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:25,858 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40c691bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:25,860 DEBUG [hconnection-0x6687a59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:25,863 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56490, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:25,867 DEBUG [hconnection-0x6687a59-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:25,869 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46166, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:25,984 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:26,130 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:26,134 DEBUG [hconnection-0x6687a59-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:26,139 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42000, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:26,268 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:26,484 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:26,600 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:27,885 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:27,885 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:27,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:27,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 10:00:27,890 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109627890"}]},"ts":"1676109627890"} 2023-02-11 10:00:27,892 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:27,894 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:27,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:27,896 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:27,896 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:27,896 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:27,897 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:27,897 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:27,897 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:27,897 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109627897"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627897"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627897"}]},"ts":"1676109627897"} 2023-02-11 10:00:27,898 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:27,898 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:27,898 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:27,898 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:27,898 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109627898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627898"}]},"ts":"1676109627898"} 2023-02-11 10:00:27,898 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109627898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627898"}]},"ts":"1676109627898"} 2023-02-11 10:00:27,898 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109627898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627898"}]},"ts":"1676109627898"} 2023-02-11 10:00:27,898 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109627898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627898"}]},"ts":"1676109627898"} 2023-02-11 10:00:27,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=79, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:27,900 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=82, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:27,901 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=80, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:27,902 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=78, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:27,903 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:27,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=77, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:27,904 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:27,904 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109627904"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109627904"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109627904"}]},"ts":"1676109627904"} 2023-02-11 10:00:27,906 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:28,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:28,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:28,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:28,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:28,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:28,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:28,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 86a1312865cab1cc55298ab2c74451da 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-11 10:00:28,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:28,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:28,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:28,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:28,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:28,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:28,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4f728331505b601d10a557ad43850320 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-11 10:00:28,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:28,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:28,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:28,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:28,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:28,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:28,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1bff2e862183c9fd81aa3c886062178c 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-11 10:00:28,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:28,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:28,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/.tmp/cf/98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:28,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/.tmp/cf/7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:28,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/.tmp/cf/98464fb3faf34d1ba2a19a37143ccc6f as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:28,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/.tmp/cf/7272f33cb3994cf39cac4f156ba82a23 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:28,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f, entries=31, sequenceid=15, filesize=10.2 K 2023-02-11 10:00:28,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 86a1312865cab1cc55298ab2c74451da in 68ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23, entries=31, sequenceid=15, filesize=10.2 K 2023-02-11 10:00:28,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 4f728331505b601d10a557ad43850320 in 71ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,132 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:28,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109628052Running coprocessor pre-close hooks at 1676109628052Disabling compacts and flushes for region at 1676109628052Disabling writes for close at 1676109628053 (+1 ms)Obtaining lock to block concurrent updates at 1676109628054 (+1 ms)Preparing flush snapshotting stores in 86a1312865cab1cc55298ab2c74451da at 1676109628054Finished memstore snapshotting testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676109628054Flushing stores of testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. at 1676109628058 (+4 ms)Flushing 86a1312865cab1cc55298ab2c74451da/cf: creating writer at 1676109628060 (+2 ms)Flushing mob file 86a1312865cab1cc55298ab2c74451da/cf: appending metadata at 1676109628078 (+18 ms)Flushing mob file 86a1312865cab1cc55298ab2c74451da/cf: closing flushed file at 1676109628078Flushing 86a1312865cab1cc55298ab2c74451da/cf: appending metadata at 1676109628104 (+26 ms)Flushing 86a1312865cab1cc55298ab2c74451da/cf: closing flushed file at 1676109628104Flushing 86a1312865cab1cc55298ab2c74451da/cf: reopening flushed file at 1676109628116 (+12 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 86a1312865cab1cc55298ab2c74451da in 68ms, sequenceid=15, compaction requested=false at 1676109628121 (+5 ms)Writing region close event to WAL at 1676109628125 (+4 ms)Running coprocessor post-close hooks at 1676109628133 (+8 ms)Closed at 1676109628133 2023-02-11 10:00:28,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:28,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:28,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:28,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:28,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:28,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:28,136 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:28,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:28,136 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109628136"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628136"}]},"ts":"1676109628136"} 2023-02-11 10:00:28,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 59ce432d7225c2fa47a6ee02f27bbfdc 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-11 10:00:28,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:28,136 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109628054Running coprocessor pre-close hooks at 1676109628054Disabling compacts and flushes for region at 1676109628054Disabling writes for close at 1676109628055 (+1 ms)Obtaining lock to block concurrent updates at 1676109628056 (+1 ms)Preparing flush snapshotting stores in 4f728331505b601d10a557ad43850320 at 1676109628056Finished memstore snapshotting testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676109628056Flushing stores of testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. at 1676109628057 (+1 ms)Flushing 4f728331505b601d10a557ad43850320/cf: creating writer at 1676109628058 (+1 ms)Flushing mob file 4f728331505b601d10a557ad43850320/cf: appending metadata at 1676109628078 (+20 ms)Flushing mob file 4f728331505b601d10a557ad43850320/cf: closing flushed file at 1676109628078Flushing 4f728331505b601d10a557ad43850320/cf: appending metadata at 1676109628104 (+26 ms)Flushing 4f728331505b601d10a557ad43850320/cf: closing flushed file at 1676109628104Flushing 4f728331505b601d10a557ad43850320/cf: reopening flushed file at 1676109628119 (+15 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 4f728331505b601d10a557ad43850320 in 71ms, sequenceid=15, compaction requested=false at 1676109628126 (+7 ms)Writing region close event to WAL at 1676109628129 (+3 ms)Running coprocessor post-close hooks at 1676109628136 (+7 ms)Closed at 1676109628136 2023-02-11 10:00:28,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:28,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:28,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:28,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:28,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:28,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:28,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:28,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing b714c90af14d8acfa1b06ca5768827da 1/1 column families, dataSize=22.31 KB heapSize=48.20 KB 2023-02-11 10:00:28,142 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:28,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 10:00:28,142 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109628142"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628142"}]},"ts":"1676109628142"} 2023-02-11 10:00:28,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-11 10:00:28,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 232 msec 2023-02-11 10:00:28,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 255 msec 2023-02-11 10:00:28,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=78 2023-02-11 10:00:28,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=78, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 247 msec 2023-02-11 10:00:28,153 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 257 msec 2023-02-11 10:00:28,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:28,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:28,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/.tmp/cf/7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:28,188 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/.tmp/cf/7de4e72b64f140619c53542c482912d0 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:28,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/.tmp/cf/21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:28,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0, entries=35, sequenceid=15, filesize=10.9 K 2023-02-11 10:00:28,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/.tmp/cf/21dc81187d984457854aefc08e61f8d8 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:28,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 59ce432d7225c2fa47a6ee02f27bbfdc in 60ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8, entries=341, sequenceid=15, filesize=63.0 K 2023-02-11 10:00:28,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for b714c90af14d8acfa1b06ca5768827da in 63ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:28,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109628135Running coprocessor pre-close hooks at 1676109628135Disabling compacts and flushes for region at 1676109628135Disabling writes for close at 1676109628136 (+1 ms)Obtaining lock to block concurrent updates at 1676109628136Preparing flush snapshotting stores in 59ce432d7225c2fa47a6ee02f27bbfdc at 1676109628136Finished memstore snapshotting testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676109628136Flushing stores of testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. at 1676109628137 (+1 ms)Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: creating writer at 1676109628137Flushing mob file 59ce432d7225c2fa47a6ee02f27bbfdc/cf: appending metadata at 1676109628149 (+12 ms)Flushing mob file 59ce432d7225c2fa47a6ee02f27bbfdc/cf: closing flushed file at 1676109628149Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: appending metadata at 1676109628172 (+23 ms)Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: closing flushed file at 1676109628172Flushing 59ce432d7225c2fa47a6ee02f27bbfdc/cf: reopening flushed file at 1676109628189 (+17 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 59ce432d7225c2fa47a6ee02f27bbfdc in 60ms, sequenceid=15, compaction requested=false at 1676109628196 (+7 ms)Writing region close event to WAL at 1676109628200 (+4 ms)Running coprocessor post-close hooks at 1676109628212 (+12 ms)Closed at 1676109628212 2023-02-11 10:00:28,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:28,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109628141Running coprocessor pre-close hooks at 1676109628141Disabling compacts and flushes for region at 1676109628141Disabling writes for close at 1676109628142 (+1 ms)Obtaining lock to block concurrent updates at 1676109628142Preparing flush snapshotting stores in b714c90af14d8acfa1b06ca5768827da at 1676109628142Finished memstore snapshotting testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., syncing WAL and waiting on mvcc, flushsize=dataSize=22843, getHeapSize=49344, getOffHeapSize=0, getCellsCount=341 at 1676109628142Flushing stores of testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. at 1676109628148 (+6 ms)Flushing b714c90af14d8acfa1b06ca5768827da/cf: creating writer at 1676109628149 (+1 ms)Flushing mob file b714c90af14d8acfa1b06ca5768827da/cf: appending metadata at 1676109628160 (+11 ms)Flushing mob file b714c90af14d8acfa1b06ca5768827da/cf: closing flushed file at 1676109628160Flushing b714c90af14d8acfa1b06ca5768827da/cf: appending metadata at 1676109628178 (+18 ms)Flushing b714c90af14d8acfa1b06ca5768827da/cf: closing flushed file at 1676109628178Flushing b714c90af14d8acfa1b06ca5768827da/cf: reopening flushed file at 1676109628194 (+16 ms)Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for b714c90af14d8acfa1b06ca5768827da in 63ms, sequenceid=15, compaction requested=false at 1676109628205 (+11 ms)Writing region close event to WAL at 1676109628209 (+4 ms)Running coprocessor post-close hooks at 1676109628213 (+4 ms)Closed at 1676109628213 2023-02-11 10:00:28,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:28,214 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:28,215 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109628214"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628214"}]},"ts":"1676109628214"} 2023-02-11 10:00:28,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:28,215 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:28,215 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109628215"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628215"}]},"ts":"1676109628215"} 2023-02-11 10:00:28,218 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=79 2023-02-11 10:00:28,218 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=79, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 317 msec 2023-02-11 10:00:28,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=82 2023-02-11 10:00:28,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=82, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 317 msec 2023-02-11 10:00:28,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 324 msec 2023-02-11 10:00:28,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 325 msec 2023-02-11 10:00:28,297 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:28,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:28,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/.tmp/cf/843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:28,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/.tmp/cf/843d3d9439ae414e8d30988488444e12 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:28,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12, entries=32, sequenceid=15, filesize=10.4 K 2023-02-11 10:00:28,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1bff2e862183c9fd81aa3c886062178c in 464ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:28,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109628056Running coprocessor pre-close hooks at 1676109628056Disabling compacts and flushes for region at 1676109628056Disabling writes for close at 1676109628056Obtaining lock to block concurrent updates at 1676109628056Preparing flush snapshotting stores in 1bff2e862183c9fd81aa3c886062178c at 1676109628056Finished memstore snapshotting testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676109628056Flushing stores of testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. at 1676109628059 (+3 ms)Flushing 1bff2e862183c9fd81aa3c886062178c/cf: creating writer at 1676109628060 (+1 ms)Flushing mob file 1bff2e862183c9fd81aa3c886062178c/cf: appending metadata at 1676109628077 (+17 ms)Flushing mob file 1bff2e862183c9fd81aa3c886062178c/cf: closing flushed file at 1676109628077Flushing 1bff2e862183c9fd81aa3c886062178c/cf: appending metadata at 1676109628504 (+427 ms)Flushing 1bff2e862183c9fd81aa3c886062178c/cf: closing flushed file at 1676109628504Flushing 1bff2e862183c9fd81aa3c886062178c/cf: reopening flushed file at 1676109628516 (+12 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1bff2e862183c9fd81aa3c886062178c in 464ms, sequenceid=15, compaction requested=false at 1676109628520 (+4 ms)Writing region close event to WAL at 1676109628523 (+3 ms)Running coprocessor post-close hooks at 1676109628528 (+5 ms)Closed at 1676109628529 (+1 ms) 2023-02-11 10:00:28,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:28,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:28,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:28,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:28,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:28,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:28,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:28,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing eb9c4974f7474c0e09c82c0ecda0c585 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-11 10:00:28,532 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:28,532 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109628532"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628532"}]},"ts":"1676109628532"} 2023-02-11 10:00:28,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=80 2023-02-11 10:00:28,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=80, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,35443,1676109614327 in 634 msec 2023-02-11 10:00:28,538 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 643 msec 2023-02-11 10:00:28,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:28,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/.tmp/cf/509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:28,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/.tmp/cf/509bed9065f84d0ba31ce69a3a3f4bd0 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:28,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0, entries=30, sequenceid=15, filesize=10.0 K 2023-02-11 10:00:28,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for eb9c4974f7474c0e09c82c0ecda0c585 in 38ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:28,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:28,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:28,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109628531Running coprocessor pre-close hooks at 1676109628531Disabling compacts and flushes for region at 1676109628531Disabling writes for close at 1676109628532 (+1 ms)Obtaining lock to block concurrent updates at 1676109628532Preparing flush snapshotting stores in eb9c4974f7474c0e09c82c0ecda0c585 at 1676109628532Finished memstore snapshotting testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676109628532Flushing stores of testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. at 1676109628533 (+1 ms)Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: creating writer at 1676109628534 (+1 ms)Flushing mob file eb9c4974f7474c0e09c82c0ecda0c585/cf: appending metadata at 1676109628538 (+4 ms)Flushing mob file eb9c4974f7474c0e09c82c0ecda0c585/cf: closing flushed file at 1676109628538Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: appending metadata at 1676109628552 (+14 ms)Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: closing flushed file at 1676109628552Flushing eb9c4974f7474c0e09c82c0ecda0c585/cf: reopening flushed file at 1676109628566 (+14 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for eb9c4974f7474c0e09c82c0ecda0c585 in 38ms, sequenceid=15, compaction requested=false at 1676109628571 (+5 ms)Writing region close event to WAL at 1676109628575 (+4 ms)Running coprocessor post-close hooks at 1676109628581 (+6 ms)Closed at 1676109628581 2023-02-11 10:00:28,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:28,584 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:28,584 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109628584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109628584"}]},"ts":"1676109628584"} 2023-02-11 10:00:28,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=77 2023-02-11 10:00:28,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=77, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 683 msec 2023-02-11 10:00:28,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=76 2023-02-11 10:00:28,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 694 msec 2023-02-11 10:00:28,590 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109628590"}]},"ts":"1676109628590"} 2023-02-11 10:00:28,592 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:28,594 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:28,596 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 710 msec 2023-02-11 10:00:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-11 10:00:28,644 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 76 completed 2023-02-11 10:00:28,646 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109628646 (current time:1676109628646). 2023-02-11 10:00:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x692df70e to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c85d62b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:28,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:28,660 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56502, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:28,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x692df70e to 127.0.0.1:64863 2023-02-11 10:00:28,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:28,669 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:28,670 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110228670 2023-02-11 10:00:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,670 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:28,670 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:28,670 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:28,691 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:28,696 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,696 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,696 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,696 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,696 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,697 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,699 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,700 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,700 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,702 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,702 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,703 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,703 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,704 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7] hfiles 2023-02-11 10:00:28,704 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:28,705 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:28,705 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f] hfiles 2023-02-11 10:00:28,705 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:28,705 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8] hfiles 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4] hfiles 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:28,707 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:28,711 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823] hfiles 2023-02-11 10:00:28,711 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:28,711 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:28,711 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:28,712 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:28,721 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f] hfiles 2023-02-11 10:00:28,721 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:28,721 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:28,721 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c] hfiles 2023-02-11 10:00:28,721 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:28,722 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:28,787 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:28,787 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676109617344 2023-02-11 10:00:28,788 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb1-1676109617344 2023-02-11 10:00:28,836 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb1-1676109617344 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/snaptb1-1676109617344 2023-02-11 10:00:28,856 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676109617344 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:28,857 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb1-1676109617344 2023-02-11 10:00:28,857 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb1-1676109617344 2023-02-11 10:00:28,857 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676109617344 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109628670Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109628691 (+21 ms)Consolidate snapshot: snaptb1-1676109617344 at 1676109628787 (+96 ms)Loading Region manifests for snaptb1-1676109617344 at 1676109628787Writing data manifest for snaptb1-1676109617344 at 1676109628798 (+11 ms)Verifying snapshot: snaptb1-1676109617344 at 1676109628814 (+16 ms)Snapshot snaptb1-1676109617344 of table testRestoreSnapshot-1676109617344 completed at 1676109628856 (+42 ms) 2023-02-11 10:00:28,859 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:28,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 194 msec 2023-02-11 10:00:28,920 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:28,937 INFO [Listener at localhost/42305] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:28,937 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109628937 (current time:1676109628937). 2023-02-11 10:00:28,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x784aeef0 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@641b9b5d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:28,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:28,946 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x784aeef0 to 127.0.0.1:64863 2023-02-11 10:00:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:28,951 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:28,952 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110228952 2023-02-11 10:00:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:28,952 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676109617344-1676109628936 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:28,952 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:28,952 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:28,963 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,965 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:28,967 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8] hfiles 2023-02-11 10:00:28,967 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:28,967 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:28,968 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7] hfiles 2023-02-11 10:00:28,968 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:28,968 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:28,968 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c] hfiles 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f] hfiles 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f] hfiles 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:28,969 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:28,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:28,970 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4] hfiles 2023-02-11 10:00:28,970 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:28,970 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:28,973 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823] hfiles 2023-02-11 10:00:28,973 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:28,973 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:28,973 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:28,973 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:28,974 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:29,008 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:29,008 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,009 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,043 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676109617344-1676109628936 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,061 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676109617344-1676109628936 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:29,061 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,061 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,061 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676109617344-1676109628936 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109628952Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109628963 (+11 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676109617344-1676109628936 at 1676109629008 (+45 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676109617344-1676109628936 at 1676109629009 (+1 ms)Writing data manifest for hbase-failsafe-snaptb0-1676109617344-1676109628936 at 1676109629016 (+7 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676109617344-1676109628936 at 1676109629026 (+10 ms)Snapshot hbase-failsafe-snaptb0-1676109617344-1676109628936 of table testRestoreSnapshot-1676109617344 completed at 1676109629061 (+35 ms) 2023-02-11 10:00:29,063 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:29,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 113 msec 2023-02-11 10:00:29,203 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:29,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:29,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676109617344-1676109628936 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb0-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109621691 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 10:00:29,239 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676109617344 as table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 10:00:29,260 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000002 2023-02-11 10:00:29,261 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:29,261 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000002 2023-02-11 10:00:29,262 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:29,266 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109621691 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:29,266 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,277 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,292 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:29,294 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:29,296 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:29,298 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:29,303 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:29,305 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:29,305 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,305 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,305 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,306 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,306 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,306 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,323 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:29,324 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:29,325 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:29,326 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:29,326 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:29,327 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:29,327 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109621691 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,330 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109629329"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:29,335 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,335 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,336 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629330"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109629330"}]},"ts":"1676109629330"} 2023-02-11 10:00:29,338 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:29,338 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 10:00:29,338 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:29,339 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676109617344 completed! 2023-02-11 10:00:29,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb0-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109621691 type: DISABLED version: 2 ttl: 0 ) in 113 msec 2023-02-11 10:00:29,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-11 10:00:29,495 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676109617344, procId: 91 completed 2023-02-11 10:00:29,495 INFO [Listener at localhost/42305] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,499 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676109617344-1676109628936" 2023-02-11 10:00:29,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676109617344-1676109628936 2023-02-11 10:00:29,503 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,504 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 10:00:29,508 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676109617344 2023-02-11 10:00:29,508 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109629508"}]},"ts":"1676109629508"} 2023-02-11 10:00:29,509 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:29,512 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:29,514 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:29,516 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:29,516 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:29,516 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:29,517 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:29,517 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,517 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,517 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,517 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,518 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:29,518 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:29,519 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,519 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:29,668 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:29,668 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:29,668 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:29,668 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:29,668 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:29,671 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:29,671 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:29,671 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629671"}]},"ts":"1676109629671"} 2023-02-11 10:00:29,671 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:29,671 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:29,671 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:29,672 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629671"}]},"ts":"1676109629671"} 2023-02-11 10:00:29,672 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629671"}]},"ts":"1676109629671"} 2023-02-11 10:00:29,672 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629671"}]},"ts":"1676109629671"} 2023-02-11 10:00:29,672 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629671"}]},"ts":"1676109629671"} 2023-02-11 10:00:29,673 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=97, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:29,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=93, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:29,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=96, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:29,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=94, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:29,677 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:29,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=98, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:29,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629677"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109629677"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109629677"}]},"ts":"1676109629677"} 2023-02-11 10:00:29,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 10:00:29,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:29,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:29,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,831 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,831 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:29,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:29,832 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:29,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,832 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:29,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,832 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:29,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,832 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:29,832 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:29,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,833 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:29,834 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,835 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:29,835 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:29,835 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:29,840 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:29,840 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:29,842 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,842 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,846 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:29,846 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:29,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62249157, jitterRate=-0.07241527736186981}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109629830Writing region info on filesystem at 1676109629830Initializing all the Stores at 1676109629830Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629830Cleaning up temporary data from old regions at 1676109629844 (+14 ms)Cleaning up detritus from prior splits at 1676109629845 (+1 ms)Running coprocessor post-open hooks at 1676109629848 (+3 ms)Region opened successfully at 1676109629848 2023-02-11 10:00:29,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=100, masterSystemTime=1676109629826 2023-02-11 10:00:29,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:29,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:29,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:29,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:29,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:29,852 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:29,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,852 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629852"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629852"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629852"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629852"}]},"ts":"1676109629852"} 2023-02-11 10:00:29,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58829028, jitterRate=-0.1233791708946228}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109629829Writing region info on filesystem at 1676109629829Initializing all the Stores at 1676109629830 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629830Cleaning up temporary data from old regions at 1676109629848 (+18 ms)Cleaning up detritus from prior splits at 1676109629849 (+1 ms)Running coprocessor post-open hooks at 1676109629852 (+3 ms)Region opened successfully at 1676109629852 2023-02-11 10:00:29,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:29,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=102, masterSystemTime=1676109629825 2023-02-11 10:00:29,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71405682, jitterRate=0.0640275776386261}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109629833Writing region info on filesystem at 1676109629833Initializing all the Stores at 1676109629833Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629833Cleaning up temporary data from old regions at 1676109629849 (+16 ms)Cleaning up detritus from prior splits at 1676109629850 (+1 ms)Running coprocessor post-open hooks at 1676109629855 (+5 ms)Region opened successfully at 1676109629855 2023-02-11 10:00:29,855 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=103, masterSystemTime=1676109629829 2023-02-11 10:00:29,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:29,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:29,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:29,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:29,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,857 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,857 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629857"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629857"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629857"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629857"}]},"ts":"1676109629857"} 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:29,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:29,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:29,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=93 2023-02-11 10:00:29,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:29,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,858 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=93, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244 in 180 msec 2023-02-11 10:00:29,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,858 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:29,858 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109629858"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629858"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629858"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629858"}]},"ts":"1676109629858"} 2023-02-11 10:00:29,859 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,859 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,860 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 346 msec 2023-02-11 10:00:29,861 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:29,861 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:29,861 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:29,861 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:29,862 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:29,862 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:29,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=94 2023-02-11 10:00:29,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=94, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 184 msec 2023-02-11 10:00:29,863 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=98 2023-02-11 10:00:29,863 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=98, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 184 msec 2023-02-11 10:00:29,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 350 msec 2023-02-11 10:00:29,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 349 msec 2023-02-11 10:00:29,865 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:29,865 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:29,865 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:29,872 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:29,875 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,876 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:29,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,876 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,882 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:29,882 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:29,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:29,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:29,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60299705, jitterRate=-0.10146437585353851}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109629858Writing region info on filesystem at 1676109629858Initializing all the Stores at 1676109629859 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629859Cleaning up temporary data from old regions at 1676109629882 (+23 ms)Cleaning up detritus from prior splits at 1676109629883 (+1 ms)Running coprocessor post-open hooks at 1676109629886 (+3 ms)Region opened successfully at 1676109629886 2023-02-11 10:00:29,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=104, masterSystemTime=1676109629829 2023-02-11 10:00:29,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67586530, jitterRate=0.007117778062820435}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109629857Writing region info on filesystem at 1676109629857Initializing all the Stores at 1676109629858 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629858Cleaning up temporary data from old regions at 1676109629881 (+23 ms)Cleaning up detritus from prior splits at 1676109629883 (+2 ms)Running coprocessor post-open hooks at 1676109629887 (+4 ms)Region opened successfully at 1676109629887 2023-02-11 10:00:29,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=99, masterSystemTime=1676109629825 2023-02-11 10:00:29,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:29,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:29,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:29,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66372196, jitterRate=-0.010977208614349365}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:29,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109629853Writing region info on filesystem at 1676109629853Initializing all the Stores at 1676109629854 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109629854Cleaning up temporary data from old regions at 1676109629885 (+31 ms)Cleaning up detritus from prior splits at 1676109629886 (+1 ms)Running coprocessor post-open hooks at 1676109629889 (+3 ms)Region opened successfully at 1676109629889 2023-02-11 10:00:29,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=101, masterSystemTime=1676109629826 2023-02-11 10:00:29,890 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:29,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:29,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:29,890 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629890"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629890"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629890"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629890"}]},"ts":"1676109629890"} 2023-02-11 10:00:29,892 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:29,892 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629891"}]},"ts":"1676109629891"} 2023-02-11 10:00:29,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:29,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:29,893 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:29,893 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109629893"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109629893"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109629893"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109629893"}]},"ts":"1676109629893"} 2023-02-11 10:00:29,895 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-11 10:00:29,896 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327 in 213 msec 2023-02-11 10:00:29,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=96 2023-02-11 10:00:29,898 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 382 msec 2023-02-11 10:00:29,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=96, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 220 msec 2023-02-11 10:00:29,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=97 2023-02-11 10:00:29,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=97, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 223 msec 2023-02-11 10:00:29,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 384 msec 2023-02-11 10:00:29,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=92 2023-02-11 10:00:29,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 385 msec 2023-02-11 10:00:29,903 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109629903"}]},"ts":"1676109629903"} 2023-02-11 10:00:29,905 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:29,907 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676109617344' was successfully enabled. 2023-02-11 10:00:29,909 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676109617344 in 403 msec 2023-02-11 10:00:30,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-11 10:00:30,259 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 92 completed 2023-02-11 10:00:30,259 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x116b3a18 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:30,265 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5587b766, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:30,267 DEBUG [hconnection-0x4251d0a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:30,270 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56518, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:30,274 DEBUG [hconnection-0x4251d0a5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:30,277 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42016, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:30,372 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:30,496 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:30,499 DEBUG [hconnection-0x4251d0a5-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:30,501 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:30,587 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:30,728 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:30,851 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:32,149 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,149 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 10:00:32,157 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109632157"}]},"ts":"1676109632157"} 2023-02-11 10:00:32,158 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:32,163 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:32,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:32,166 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:32,166 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:32,166 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:32,166 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:32,167 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:32,167 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:32,167 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632167"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632167"}]},"ts":"1676109632167"} 2023-02-11 10:00:32,168 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:32,168 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632168"}]},"ts":"1676109632168"} 2023-02-11 10:00:32,168 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:32,168 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632168"}]},"ts":"1676109632168"} 2023-02-11 10:00:32,168 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:32,169 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632168"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632168"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632168"}]},"ts":"1676109632168"} 2023-02-11 10:00:32,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=111, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:32,169 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:32,169 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632169"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632169"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632169"}]},"ts":"1676109632169"} 2023-02-11 10:00:32,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=110, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:32,174 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=109, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:32,175 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=108, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:32,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=107, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:32,177 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:32,178 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:32,178 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632178"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109632178"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109632178"}]},"ts":"1676109632178"} 2023-02-11 10:00:32,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:32,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:32,325 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:32,326 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:32,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:32,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:32,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:32,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:32,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:32,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:32,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:32,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:32,331 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:32,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:32,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:32,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109632325Running coprocessor pre-close hooks at 1676109632325Disabling compacts and flushes for region at 1676109632325Disabling writes for close at 1676109632326 (+1 ms)Writing region close event to WAL at 1676109632328 (+2 ms)Running coprocessor post-close hooks at 1676109632341 (+13 ms)Closed at 1676109632341 2023-02-11 10:00:32,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109632328Running coprocessor pre-close hooks at 1676109632328Disabling compacts and flushes for region at 1676109632328Disabling writes for close at 1676109632330 (+2 ms)Writing region close event to WAL at 1676109632336 (+6 ms)Running coprocessor post-close hooks at 1676109632341 (+5 ms)Closed at 1676109632341 2023-02-11 10:00:32,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:32,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109632330Running coprocessor pre-close hooks at 1676109632330Disabling compacts and flushes for region at 1676109632330Disabling writes for close at 1676109632330Writing region close event to WAL at 1676109632336 (+6 ms)Running coprocessor post-close hooks at 1676109632341 (+5 ms)Closed at 1676109632341 2023-02-11 10:00:32,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:32,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:32,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:32,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:32,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:32,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:32,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:32,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:32,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:32,344 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:32,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:32,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:32,344 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632344"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632344"}]},"ts":"1676109632344"} 2023-02-11 10:00:32,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:32,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:32,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:32,344 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:32,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632344"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632344"}]},"ts":"1676109632344"} 2023-02-11 10:00:32,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:32,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:32,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:32,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:32,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:32,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:32,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:32,346 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:32,346 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632346"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632346"}]},"ts":"1676109632346"} 2023-02-11 10:00:32,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=111 2023-02-11 10:00:32,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=111, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 178 msec 2023-02-11 10:00:32,351 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-11 10:00:32,351 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244 in 168 msec 2023-02-11 10:00:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,353 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=107 2023-02-11 10:00:32,353 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=107, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 173 msec 2023-02-11 10:00:32,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 187 msec 2023-02-11 10:00:32,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:32,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-11 10:00:32,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109632343Running coprocessor pre-close hooks at 1676109632343Disabling compacts and flushes for region at 1676109632343Disabling writes for close at 1676109632343Writing region close event to WAL at 1676109632345 (+2 ms)Running coprocessor post-close hooks at 1676109632354 (+9 ms)Closed at 1676109632354 2023-02-11 10:00:32,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:32,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109632345Running coprocessor pre-close hooks at 1676109632345Disabling compacts and flushes for region at 1676109632345Disabling writes for close at 1676109632345Writing region close event to WAL at 1676109632346 (+1 ms)Running coprocessor post-close hooks at 1676109632355 (+9 ms)Closed at 1676109632355 2023-02-11 10:00:32,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:32,356 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109632344Running coprocessor pre-close hooks at 1676109632344Disabling compacts and flushes for region at 1676109632344Disabling writes for close at 1676109632344Writing region close event to WAL at 1676109632346 (+2 ms)Running coprocessor post-close hooks at 1676109632356 (+10 ms)Closed at 1676109632356 2023-02-11 10:00:32,356 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 189 msec 2023-02-11 10:00:32,357 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 190 msec 2023-02-11 10:00:32,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:32,357 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:32,357 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632357"}]},"ts":"1676109632357"} 2023-02-11 10:00:32,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:32,358 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:32,358 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632358"}]},"ts":"1676109632358"} 2023-02-11 10:00:32,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:32,358 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:32,359 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632358"}]},"ts":"1676109632358"} 2023-02-11 10:00:32,360 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=108 2023-02-11 10:00:32,361 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=108, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327 in 184 msec 2023-02-11 10:00:32,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=110 2023-02-11 10:00:32,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=110, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 189 msec 2023-02-11 10:00:32,362 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 198 msec 2023-02-11 10:00:32,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=109 2023-02-11 10:00:32,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=109, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 186 msec 2023-02-11 10:00:32,363 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 198 msec 2023-02-11 10:00:32,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=109, resume processing ppid=105 2023-02-11 10:00:32,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 199 msec 2023-02-11 10:00:32,365 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109632365"}]},"ts":"1676109632365"} 2023-02-11 10:00:32,368 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:32,370 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:32,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 221 msec 2023-02-11 10:00:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-11 10:00:32,408 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 105 completed 2023-02-11 10:00:32,417 INFO [Listener at localhost/42305] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,418 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109632418 (current time:1676109632418). 2023-02-11 10:00:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:32,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x062ae09a to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:32,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70015b64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:32,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:32,429 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56522, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062ae09a to 127.0.0.1:64863 2023-02-11 10:00:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:32,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:32,433 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:32,434 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110232434 2023-02-11 10:00:32,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:32,434 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676109617344-1676109632417 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,434 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:32,435 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:32,450 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:32,453 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,453 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,453 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,453 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,454 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,454 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,454 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,454 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,455 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,455 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,455 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,455 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,457 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f] hfiles 2023-02-11 10:00:32,457 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:32,458 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:32,458 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:32,463 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8] hfiles 2023-02-11 10:00:32,463 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:32,463 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4] hfiles 2023-02-11 10:00:32,463 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:32,464 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f] hfiles 2023-02-11 10:00:32,464 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:32,464 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7] hfiles 2023-02-11 10:00:32,464 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:32,465 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c] hfiles 2023-02-11 10:00:32,465 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa] hfiles 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:32,467 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:32,563 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:32,563 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,563 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,588 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676109617344-1676109632417 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,607 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676109617344-1676109632417 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:32,607 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,607 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,608 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676109617344-1676109632417 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109632434Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109632450 (+16 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676109617344-1676109632417 at 1676109632563 (+113 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676109617344-1676109632417 at 1676109632563Writing data manifest for hbase-failsafe-emptySnaptb-1676109617344-1676109632417 at 1676109632567 (+4 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676109617344-1676109632417 at 1676109632578 (+11 ms)Snapshot hbase-failsafe-emptySnaptb-1676109617344-1676109632417 of table testRestoreSnapshot-1676109617344 completed at 1676109632607 (+29 ms) 2023-02-11 10:00:32,609 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:32,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 178 msec 2023-02-11 10:00:32,685 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:32,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676109617344-1676109632417 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "emptySnaptb-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109618986 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 10:00:32,692 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676109617344 as table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 10:00:32,708 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000003 2023-02-11 10:00:32,709 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000002 2023-02-11 10:00:32,709 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000003 2023-02-11 10:00:32,709 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:32,711 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109618986 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:32,711 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,719 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,723 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:32,724 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:32,726 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:32,727 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:32,730 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:32,731 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 4f728331505b601d10a557ad43850320 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:32,732 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:32,746 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:32,746 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:32,746 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:32,747 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/3719691d68774d919d12001277aeabf7 2023-02-11 10:00:32,747 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:32,748 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:32,748 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109618986 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,750 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109632749"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:32,752 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109632750"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109632750"}]},"ts":"1676109632750"} 2023-02-11 10:00:32,757 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:32,757 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 10:00:32,757 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:32,757 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676109617344 completed! 2023-02-11 10:00:32,760 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "emptySnaptb-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109618986 type: DISABLED version: 2 ttl: 0 ) in 67 msec 2023-02-11 10:00:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-11 10:00:32,945 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676109617344, procId: 119 completed 2023-02-11 10:00:32,945 INFO [Listener at localhost/42305] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,946 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676109617344-1676109632417" 2023-02-11 10:00:32,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676109617344-1676109632417 2023-02-11 10:00:32,948 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,949 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 10:00:32,953 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676109617344 2023-02-11 10:00:32,953 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109632953"}]},"ts":"1676109632953"} 2023-02-11 10:00:32,954 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:32,957 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:32,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:32,959 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:32,959 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:32,959 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:32,959 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:32,960 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:32,960 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:32,960 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:32,960 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:32,960 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:32,961 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:32,961 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:32,962 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:33,110 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:33,111 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:33,111 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:33,111 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:33,111 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:33,115 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,115 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,115 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,115 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,115 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,116 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633115"}]},"ts":"1676109633115"} 2023-02-11 10:00:33,116 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633115"}]},"ts":"1676109633115"} 2023-02-11 10:00:33,115 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633115"}]},"ts":"1676109633115"} 2023-02-11 10:00:33,115 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633115"}]},"ts":"1676109633115"} 2023-02-11 10:00:33,115 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633115"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633115"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633115"}]},"ts":"1676109633115"} 2023-02-11 10:00:33,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=125, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:33,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=123, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:33,119 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:33,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=121, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:33,120 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,120 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633120"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633120"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633120"}]},"ts":"1676109633120"} 2023-02-11 10:00:33,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=124, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:33,122 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=122, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:33,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 10:00:33,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,273 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,274 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:33,275 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:33,275 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:33,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,275 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:33,276 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,277 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:33,277 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:33,277 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:33,278 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,278 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,280 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:33,280 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:33,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,280 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:33,281 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61810253, jitterRate=-0.07895545661449432}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109633272Writing region info on filesystem at 1676109633272Initializing all the Stores at 1676109633273 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633273Cleaning up temporary data from old regions at 1676109633280 (+7 ms)Cleaning up detritus from prior splits at 1676109633280Running coprocessor post-open hooks at 1676109633283 (+3 ms)Region opened successfully at 1676109633283 2023-02-11 10:00:33,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70516541, jitterRate=0.0507783442735672}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109633273Writing region info on filesystem at 1676109633273Initializing all the Stores at 1676109633274 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633274Cleaning up temporary data from old regions at 1676109633280 (+6 ms)Cleaning up detritus from prior splits at 1676109633281 (+1 ms)Running coprocessor post-open hooks at 1676109633284 (+3 ms)Region opened successfully at 1676109633284 2023-02-11 10:00:33,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=127, masterSystemTime=1676109633269 2023-02-11 10:00:33,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=128, masterSystemTime=1676109633270 2023-02-11 10:00:33,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,286 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:33,286 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633286"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633286"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633286"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633286"}]},"ts":"1676109633286"} 2023-02-11 10:00:33,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,287 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633287"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633287"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633287"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633287"}]},"ts":"1676109633287"} 2023-02-11 10:00:33,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70929266, jitterRate=0.056928426027297974}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109633275Writing region info on filesystem at 1676109633275Initializing all the Stores at 1676109633276 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633276Cleaning up temporary data from old regions at 1676109633283 (+7 ms)Cleaning up detritus from prior splits at 1676109633284 (+1 ms)Running coprocessor post-open hooks at 1676109633288 (+4 ms)Region opened successfully at 1676109633288 2023-02-11 10:00:33,288 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=131, masterSystemTime=1676109633272 2023-02-11 10:00:33,289 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,290 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:33,290 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:33,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,290 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:33,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:33,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:33,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,291 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633291"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633291"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633291"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633291"}]},"ts":"1676109633291"} 2023-02-11 10:00:33,291 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,292 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:33,292 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:33,292 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:33,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,292 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=125 2023-02-11 10:00:33,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=125, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 172 msec 2023-02-11 10:00:33,293 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,294 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:33,294 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:33,294 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=123 2023-02-11 10:00:33,294 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=123, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 172 msec 2023-02-11 10:00:33,294 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:33,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,295 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:33,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 336 msec 2023-02-11 10:00:33,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,296 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 337 msec 2023-02-11 10:00:33,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=124 2023-02-11 10:00:33,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=124, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41567,1676109614293 in 174 msec 2023-02-11 10:00:33,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73534106, jitterRate=0.0957435667514801}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109633286Writing region info on filesystem at 1676109633286Initializing all the Stores at 1676109633287 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633287Cleaning up temporary data from old regions at 1676109633294 (+7 ms)Cleaning up detritus from prior splits at 1676109633295 (+1 ms)Running coprocessor post-open hooks at 1676109633298 (+3 ms)Region opened successfully at 1676109633298 2023-02-11 10:00:33,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=132, masterSystemTime=1676109633269 2023-02-11 10:00:33,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 341 msec 2023-02-11 10:00:33,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74128504, jitterRate=0.10460078716278076}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109633288Writing region info on filesystem at 1676109633288Initializing all the Stores at 1676109633289 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633289Cleaning up temporary data from old regions at 1676109633296 (+7 ms)Cleaning up detritus from prior splits at 1676109633297 (+1 ms)Running coprocessor post-open hooks at 1676109633300 (+3 ms)Region opened successfully at 1676109633300 2023-02-11 10:00:33,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=129, masterSystemTime=1676109633270 2023-02-11 10:00:33,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,301 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70271235, jitterRate=0.04712300002574921}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:33,301 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633301"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633301"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633301"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633301"}]},"ts":"1676109633301"} 2023-02-11 10:00:33,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109633291Writing region info on filesystem at 1676109633291Initializing all the Stores at 1676109633292 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109633292Cleaning up temporary data from old regions at 1676109633298 (+6 ms)Cleaning up detritus from prior splits at 1676109633298Running coprocessor post-open hooks at 1676109633301 (+3 ms)Region opened successfully at 1676109633301 2023-02-11 10:00:33,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=130, masterSystemTime=1676109633272 2023-02-11 10:00:33,309 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,309 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633302"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633302"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633302"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633302"}]},"ts":"1676109633302"} 2023-02-11 10:00:33,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=122 2023-02-11 10:00:33,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=122, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,35443,1676109614327 in 180 msec 2023-02-11 10:00:33,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,311 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,311 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633311"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109633311"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109633311"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109633311"}]},"ts":"1676109633311"} 2023-02-11 10:00:33,311 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 353 msec 2023-02-11 10:00:33,313 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-11 10:00:33,313 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 192 msec 2023-02-11 10:00:33,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 356 msec 2023-02-11 10:00:33,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=121 2023-02-11 10:00:33,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=121, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41567,1676109614293 in 192 msec 2023-02-11 10:00:33,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=121, resume processing ppid=120 2023-02-11 10:00:33,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 357 msec 2023-02-11 10:00:33,316 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109633316"}]},"ts":"1676109633316"} 2023-02-11 10:00:33,318 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:33,319 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676109617344' was successfully enabled. 2023-02-11 10:00:33,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676109617344 in 371 msec 2023-02-11 10:00:33,636 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:33,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-11 10:00:33,704 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 120 completed 2023-02-11 10:00:33,704 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x7bfe9d88 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:33,710 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@231aba10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:33,711 DEBUG [hconnection-0x682c5896-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:33,715 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46108, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:33,719 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:33,721 DEBUG [hconnection-0x682c5896-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:33,724 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:33,724 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:33,728 DEBUG [hconnection-0x682c5896-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:33,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:33,731 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:33,734 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:33,737 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:33,746 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:33,746 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 10:00:33,749 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109633749"}]},"ts":"1676109633749"} 2023-02-11 10:00:33,751 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:33,752 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:33,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:33,756 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:33,757 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:33,757 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:33,757 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:33,757 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:33,758 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,758 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,758 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633758"}]},"ts":"1676109633758"} 2023-02-11 10:00:33,758 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,758 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633758"}]},"ts":"1676109633758"} 2023-02-11 10:00:33,758 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:33,758 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:33,758 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633758"}]},"ts":"1676109633758"} 2023-02-11 10:00:33,758 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633758"}]},"ts":"1676109633758"} 2023-02-11 10:00:33,758 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633758"}]},"ts":"1676109633758"} 2023-02-11 10:00:33,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=134, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:33,761 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=136, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:33,762 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:33,763 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:33,764 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:33,764 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=138, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:33,765 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:33,765 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633765"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109633765"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109633765"}]},"ts":"1676109633765"} 2023-02-11 10:00:33,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=137, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:33,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:33,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:33,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:33,921 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:33,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:33,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:33,926 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109633918Running coprocessor pre-close hooks at 1676109633918Disabling compacts and flushes for region at 1676109633918Disabling writes for close at 1676109633920 (+2 ms)Writing region close event to WAL at 1676109633922 (+2 ms)Running coprocessor post-close hooks at 1676109633925 (+3 ms)Closed at 1676109633926 (+1 ms) 2023-02-11 10:00:33,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:33,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,929 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:33,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109633920Running coprocessor pre-close hooks at 1676109633920Disabling compacts and flushes for region at 1676109633920Disabling writes for close at 1676109633922 (+2 ms)Writing region close event to WAL at 1676109633924 (+2 ms)Running coprocessor post-close hooks at 1676109633928 (+4 ms)Closed at 1676109633929 (+1 ms) 2023-02-11 10:00:33,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,929 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633928"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633928"}]},"ts":"1676109633928"} 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:33,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109633920Running coprocessor pre-close hooks at 1676109633920Disabling compacts and flushes for region at 1676109633920Disabling writes for close at 1676109633922 (+2 ms)Writing region close event to WAL at 1676109633924 (+2 ms)Running coprocessor post-close hooks at 1676109633928 (+4 ms)Closed at 1676109633929 (+1 ms) 2023-02-11 10:00:33,931 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:33,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:33,931 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633931"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633931"}]},"ts":"1676109633931"} 2023-02-11 10:00:33,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:33,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:33,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:33,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,933 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:33,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:33,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,934 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633933"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633933"}]},"ts":"1676109633933"} 2023-02-11 10:00:33,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:33,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,935 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=136 2023-02-11 10:00:33,935 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=136, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41193,1676109614244 in 169 msec 2023-02-11 10:00:33,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:33,937 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109633928Running coprocessor pre-close hooks at 1676109633928Disabling compacts and flushes for region at 1676109633928Disabling writes for close at 1676109633929 (+1 ms)Writing region close event to WAL at 1676109633929Running coprocessor post-close hooks at 1676109633937 (+8 ms)Closed at 1676109633937 2023-02-11 10:00:33,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=137 2023-02-11 10:00:33,940 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=137, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41567,1676109614293 in 166 msec 2023-02-11 10:00:33,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:33,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 182 msec 2023-02-11 10:00:33,944 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:33,944 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633944"}]},"ts":"1676109633944"} 2023-02-11 10:00:33,945 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 187 msec 2023-02-11 10:00:33,945 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=138 2023-02-11 10:00:33,945 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=138, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 174 msec 2023-02-11 10:00:33,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,946 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 192 msec 2023-02-11 10:00:33,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-11 10:00:33,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:33,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109633932Running coprocessor pre-close hooks at 1676109633932Disabling compacts and flushes for region at 1676109633932Disabling writes for close at 1676109633934 (+2 ms)Writing region close event to WAL at 1676109633934Running coprocessor post-close hooks at 1676109633946 (+12 ms)Closed at 1676109633947 (+1 ms) 2023-02-11 10:00:33,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:33,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109633931Running coprocessor pre-close hooks at 1676109633931Disabling compacts and flushes for region at 1676109633931Disabling writes for close at 1676109633933 (+2 ms)Writing region close event to WAL at 1676109633934 (+1 ms)Running coprocessor post-close hooks at 1676109633947 (+13 ms)Closed at 1676109633948 (+1 ms) 2023-02-11 10:00:33,948 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-11 10:00:33,949 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 182 msec 2023-02-11 10:00:33,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:33,949 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:33,949 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109633949"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633949"}]},"ts":"1676109633949"} 2023-02-11 10:00:33,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:33,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 195 msec 2023-02-11 10:00:33,950 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:33,950 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109633950"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109633950"}]},"ts":"1676109633950"} 2023-02-11 10:00:33,952 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=134 2023-02-11 10:00:33,952 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=134, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41567,1676109614293 in 191 msec 2023-02-11 10:00:33,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-02-11 10:00:33,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,35443,1676109614327 in 190 msec 2023-02-11 10:00:33,953 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 199 msec 2023-02-11 10:00:33,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=135, resume processing ppid=133 2023-02-11 10:00:33,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 200 msec 2023-02-11 10:00:33,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109633955"}]},"ts":"1676109633955"} 2023-02-11 10:00:33,956 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:33,957 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:33,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 221 msec 2023-02-11 10:00:34,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-11 10:00:34,000 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 133 completed 2023-02-11 10:00:34,012 INFO [Listener at localhost/42305] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,013 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:34,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109634013 (current time:1676109634013). 2023-02-11 10:00:34,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:34,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x5d5f6372 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:34,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35df175a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:34,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:34,024 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:34,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d5f6372 to 127.0.0.1:64863 2023-02-11 10:00:34,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:34,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:34,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:34,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:34,034 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:34,034 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE; timeout=600000, timestamp=1676110234034 2023-02-11 10:00:34,034 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676109617344-1676109634012 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } 2023-02-11 10:00:34,035 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:34,035 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:34,046 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:34,048 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,048 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,048 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,048 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,048 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,049 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:34,050 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,051 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,051 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,051 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,058 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,058 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,058 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:34,115 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:34,115 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,115 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,136 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676109617344-1676109634012 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,154 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676109617344-1676109634012 of table testRestoreSnapshot-1676109617344 completed 2023-02-11 10:00:34,155 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,155 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,155 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676109617344-1676109634012 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676109617344 at 1676109634035Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } at 1676109634046 (+11 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676109617344-1676109634012 at 1676109634115 (+69 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676109617344-1676109634012 at 1676109634115Writing data manifest for hbase-failsafe-snaptb1-1676109617344-1676109634012 at 1676109634119 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676109617344-1676109634012 at 1676109634130 (+11 ms)Snapshot hbase-failsafe-snaptb1-1676109617344-1676109634012 of table testRestoreSnapshot-1676109617344 completed at 1676109634154 (+24 ms) 2023-02-11 10:00:34,156 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE 2023-02-11 10:00:34,158 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676109617344, type=EXCLUSIVE in 125 msec 2023-02-11 10:00:34,285 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:34,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 } is done 2023-02-11 10:00:34,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676109617344-1676109634012 table=testRestoreSnapshot-1676109617344 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:34,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 10:00:34,308 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676109617344 as table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 10:00:34,321 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000004 2023-02-11 10:00:34,322 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000003 2023-02-11 10:00:34,322 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000004 2023-02-11 10:00:34,323 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } 2023-02-11 10:00:34,326 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:34,326 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,333 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,335 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,338 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,341 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,345 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,348 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,351 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,354 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,357 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,361 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,364 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,368 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,372 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 of snapshot=snaptb1-1676109617344 to region=ddb1336944af1bf0427d1343326835ea table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:34,376 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:34,384 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 843d3d9439ae414e8d30988488444e12 of snapshot=snaptb1-1676109617344 to region=1bff2e862183c9fd81aa3c886062178c table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,385 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7de4e72b64f140619c53542c482912d0 of snapshot=snaptb1-1676109617344 to region=59ce432d7225c2fa47a6ee02f27bbfdc table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,385 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 21dc81187d984457854aefc08e61f8d8 of snapshot=snaptb1-1676109617344 to region=b714c90af14d8acfa1b06ca5768827da table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,387 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7272f33cb3994cf39cac4f156ba82a23 of snapshot=snaptb1-1676109617344 to region=4f728331505b601d10a557ad43850320 table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,388 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 509bed9065f84d0ba31ce69a3a3f4bd0 of snapshot=snaptb1-1676109617344 to region=eb9c4974f7474c0e09c82c0ecda0c585 table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,389 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 98464fb3faf34d1ba2a19a37143ccc6f of snapshot=snaptb1-1676109617344 to region=86a1312865cab1cc55298ab2c74451da table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,398 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink fa1f306b33c8410e8fc7746a0d41671f of snapshot=snaptb1-1676109617344 to region=59ce432d7225c2fa47a6ee02f27bbfdc table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,400 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ca87f195a3104b539048984ba5a3a37c of snapshot=snaptb1-1676109617344 to region=86a1312865cab1cc55298ab2c74451da table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,400 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c8e02b793157422b98df2d998ef0b4c8 of snapshot=snaptb1-1676109617344 to region=eb9c4974f7474c0e09c82c0ecda0c585 table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,400 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3719691d68774d919d12001277aeabf7 of snapshot=snaptb1-1676109617344 to region=b714c90af14d8acfa1b06ca5768827da table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,401 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7558e76cf7794bd9821d73896593011f of snapshot=snaptb1-1676109617344 to region=4f728331505b601d10a557ad43850320 table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,401 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f700cba99b9d45a6939369b56232ede4 of snapshot=snaptb1-1676109617344 to region=1bff2e862183c9fd81aa3c886062178c table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,409 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,410 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109634410"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:34,412 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,413 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109634411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109634411"}]},"ts":"1676109634411"} 2023-02-11 10:00:34,415 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:34,415 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-11 10:00:34,415 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}] 2023-02-11 10:00:34,415 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676109617344 table=testRestoreSnapshot-1676109617344 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676109617344 completed! 2023-02-11 10:00:34,417 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 ) in 111 msec 2023-02-11 10:00:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-11 10:00:34,561 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676109617344, procId: 147 completed 2023-02-11 10:00:34,561 INFO [Listener at localhost/42305] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,562 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676109617344-1676109634012" 2023-02-11 10:00:34,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676109617344-1676109634012 2023-02-11 10:00:34,565 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,566 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 10:00:34,570 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676109617344 2023-02-11 10:00:34,570 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109634570"}]},"ts":"1676109634570"} 2023-02-11 10:00:34,571 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:34,574 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:34,574 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:34,576 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:34,576 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:34,576 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:34,577 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:34,578 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,578 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,578 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,578 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:34,578 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,579 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:34,579 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,580 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-11 10:00:34,700 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bfe9d88 to 127.0.0.1:64863 2023-02-11 10:00:34,700 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:34,701 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14908088 to 127.0.0.1:64863 2023-02-11 10:00:34,701 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:34,702 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x116b3a18 to 127.0.0.1:64863 2023-02-11 10:00:34,702 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:34,728 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:34,728 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:34,728 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:34,728 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:34,728 DEBUG [jenkins-hbase4:42723] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:34,732 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:34,732 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:34,732 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634732"}]},"ts":"1676109634732"} 2023-02-11 10:00:34,732 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:34,732 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:34,732 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:34,732 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634732"}]},"ts":"1676109634732"} 2023-02-11 10:00:34,732 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634732"}]},"ts":"1676109634732"} 2023-02-11 10:00:34,732 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109634732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634732"}]},"ts":"1676109634732"} 2023-02-11 10:00:34,732 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109634732"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634732"}]},"ts":"1676109634732"} 2023-02-11 10:00:34,734 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=150, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:34,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=153, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:34,735 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:34,736 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=149, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:34,737 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=154, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:34,737 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:34,737 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634737"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109634737"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109634737"}]},"ts":"1676109634737"} 2023-02-11 10:00:34,739 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 10:00:34,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:34,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:34,890 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,891 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:34,891 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:34,891 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,891 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:34,892 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:34,892 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:34,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:34,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:34,892 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:34,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,894 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,894 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:34,895 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:34,895 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:34,914 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:34,914 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:34,914 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:34,926 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:34,927 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:34,927 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:34,928 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:34,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,928 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:34,929 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:34,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:34,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61738445, jitterRate=-0.08002547919750214}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:34,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61881693, jitterRate=-0.07789091765880585}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109634888Writing region info on filesystem at 1676109634888Initializing all the Stores at 1676109634889 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634889Cleaning up temporary data from old regions at 1676109634931 (+42 ms)Cleaning up detritus from prior splits at 1676109634931Running coprocessor post-open hooks at 1676109634934 (+3 ms)Region opened successfully at 1676109634934 2023-02-11 10:00:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109634890Writing region info on filesystem at 1676109634890Initializing all the Stores at 1676109634891 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634891Cleaning up temporary data from old regions at 1676109634931 (+40 ms)Cleaning up detritus from prior splits at 1676109634931Running coprocessor post-open hooks at 1676109634934 (+3 ms)Region opened successfully at 1676109634934 2023-02-11 10:00:34,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=157, masterSystemTime=1676109634887 2023-02-11 10:00:34,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=155, masterSystemTime=1676109634885 2023-02-11 10:00:34,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70868628, jitterRate=0.05602484941482544}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:34,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109634893Writing region info on filesystem at 1676109634893Initializing all the Stores at 1676109634893Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634893Cleaning up temporary data from old regions at 1676109634932 (+39 ms)Cleaning up detritus from prior splits at 1676109634932Running coprocessor post-open hooks at 1676109634936 (+4 ms)Region opened successfully at 1676109634936 2023-02-11 10:00:34,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=160, masterSystemTime=1676109634888 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:34,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:34,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,938 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:34,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:34,938 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634938"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109634938"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109634938"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109634938"}]},"ts":"1676109634938"} 2023-02-11 10:00:34,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:34,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:34,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:34,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:34,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:34,939 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:34,940 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634939"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109634939"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109634939"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109634939"}]},"ts":"1676109634939"} 2023-02-11 10:00:34,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:34,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:34,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:34,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:34,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:34,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:34,941 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:34,941 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109634941"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109634941"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109634941"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109634941"}]},"ts":"1676109634941"} 2023-02-11 10:00:34,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-11 10:00:34,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 205 msec 2023-02-11 10:00:34,947 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=150 2023-02-11 10:00:34,947 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=150, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 208 msec 2023-02-11 10:00:34,947 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 370 msec 2023-02-11 10:00:34,948 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-11 10:00:34,948 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327 in 204 msec 2023-02-11 10:00:34,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 373 msec 2023-02-11 10:00:34,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 374 msec 2023-02-11 10:00:34,955 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:34,955 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:34,955 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:34,956 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:34,956 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:34,956 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:34,956 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:34,957 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:34,957 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:34,958 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:34,958 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:34,958 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:34,978 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:34,984 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:34,994 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:34,997 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:34,997 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:35,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:35,002 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 2023-02-11 10:00:35,002 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:35,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:35,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:35,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:35,005 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:35,006 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:35,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:35,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:35,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:35,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64024252, jitterRate=-0.04596430063247681}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:35,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109634939Writing region info on filesystem at 1676109634939Initializing all the Stores at 1676109634941 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634941Cleaning up temporary data from old regions at 1676109635003 (+62 ms)Cleaning up detritus from prior splits at 1676109635005 (+2 ms)Running coprocessor post-open hooks at 1676109635008 (+3 ms)Region opened successfully at 1676109635008 2023-02-11 10:00:35,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:35,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=158, masterSystemTime=1676109634887 2023-02-11 10:00:35,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60047255, jitterRate=-0.10522617399692535}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:35,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109634941Writing region info on filesystem at 1676109634941Initializing all the Stores at 1676109634942 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634942Cleaning up temporary data from old regions at 1676109635005 (+63 ms)Cleaning up detritus from prior splits at 1676109635006 (+1 ms)Running coprocessor post-open hooks at 1676109635009 (+3 ms)Region opened successfully at 1676109635009 2023-02-11 10:00:35,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=159, masterSystemTime=1676109634888 2023-02-11 10:00:35,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:35,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:35,011 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:35,011 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109635011"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109635011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109635011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109635011"}]},"ts":"1676109635011"} 2023-02-11 10:00:35,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:35,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:35,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:35,012 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:35,012 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109635012"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109635012"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109635012"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109635012"}]},"ts":"1676109635012"} 2023-02-11 10:00:35,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64882389, jitterRate=-0.03317706286907196}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:35,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109634941Writing region info on filesystem at 1676109634941Initializing all the Stores at 1676109634944 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109634944Cleaning up temporary data from old regions at 1676109635008 (+64 ms)Cleaning up detritus from prior splits at 1676109635009 (+1 ms)Running coprocessor post-open hooks at 1676109635013 (+4 ms)Region opened successfully at 1676109635013 2023-02-11 10:00:35,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=156, masterSystemTime=1676109634885 2023-02-11 10:00:35,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:35,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:35,015 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=149 2023-02-11 10:00:35,015 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=149, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244 in 277 msec 2023-02-11 10:00:35,016 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:35,016 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109635016"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109635016"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109635016"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109635016"}]},"ts":"1676109635016"} 2023-02-11 10:00:35,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=154 2023-02-11 10:00:35,016 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=154, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 277 msec 2023-02-11 10:00:35,017 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 442 msec 2023-02-11 10:00:35,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 442 msec 2023-02-11 10:00:35,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=153 2023-02-11 10:00:35,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=153, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 283 msec 2023-02-11 10:00:35,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=148 2023-02-11 10:00:35,027 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 445 msec 2023-02-11 10:00:35,027 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109635027"}]},"ts":"1676109635027"} 2023-02-11 10:00:35,029 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:35,031 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676109617344' was successfully enabled. 2023-02-11 10:00:35,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676109617344 in 466 msec 2023-02-11 10:00:35,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-11 10:00:35,321 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 148 completed 2023-02-11 10:00:35,322 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x29616110 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:35,329 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c92e86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:35,332 DEBUG [hconnection-0x2f9f32ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:35,338 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46126, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:35,341 DEBUG [hconnection-0x2f9f32ef-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:35,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53580, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:35,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,584 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:35,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,852 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:35,855 DEBUG [hconnection-0x2f9f32ef-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:35,859 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52852, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:35,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:35,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,143 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:36,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,489 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,766 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:36,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:36,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:37,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,776 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:38,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:39,465 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,466 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 10:00:39,474 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109639474"}]},"ts":"1676109639474"} 2023-02-11 10:00:39,475 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:39,477 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:39,478 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:39,480 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:39,480 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:39,480 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:39,480 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:39,480 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:39,481 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:39,481 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:39,481 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639481"}]},"ts":"1676109639481"} 2023-02-11 10:00:39,481 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:39,481 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639481"}]},"ts":"1676109639481"} 2023-02-11 10:00:39,481 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109639481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639481"}]},"ts":"1676109639481"} 2023-02-11 10:00:39,481 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:39,481 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:39,481 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639481"}]},"ts":"1676109639481"} 2023-02-11 10:00:39,481 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109639481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639481"}]},"ts":"1676109639481"} 2023-02-11 10:00:39,482 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=166, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:39,483 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=163, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:39,486 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=162, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:39,486 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=167, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:39,486 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:39,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=165, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:39,487 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:39,487 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639487"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109639487"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109639487"}]},"ts":"1676109639487"} 2023-02-11 10:00:39,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=164, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:39,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:39,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:39,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:39,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:39,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:39,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:39,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:39,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:39,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:39,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:39,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:39,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:39,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:39,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:39,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:39,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:39,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:39,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:39,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:39,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:39,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109639636Running coprocessor pre-close hooks at 1676109639636Disabling compacts and flushes for region at 1676109639636Disabling writes for close at 1676109639637 (+1 ms)Writing region close event to WAL at 1676109639643 (+6 ms)Running coprocessor post-close hooks at 1676109639652 (+9 ms)Closed at 1676109639653 (+1 ms) 2023-02-11 10:00:39,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109639640Running coprocessor pre-close hooks at 1676109639640Disabling compacts and flushes for region at 1676109639640Disabling writes for close at 1676109639641 (+1 ms)Writing region close event to WAL at 1676109639649 (+8 ms)Running coprocessor post-close hooks at 1676109639653 (+4 ms)Closed at 1676109639653 2023-02-11 10:00:39,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:39,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109639641Running coprocessor pre-close hooks at 1676109639641Disabling compacts and flushes for region at 1676109639641Disabling writes for close at 1676109639643 (+2 ms)Writing region close event to WAL at 1676109639649 (+6 ms)Running coprocessor post-close hooks at 1676109639654 (+5 ms)Closed at 1676109639654 2023-02-11 10:00:39,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:39,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:39,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:39,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:39,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:39,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:39,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:39,656 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:39,656 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639656"}]},"ts":"1676109639656"} 2023-02-11 10:00:39,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:39,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:39,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:39,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:39,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:39,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:39,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:39,660 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:39,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639659"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639659"}]},"ts":"1676109639659"} 2023-02-11 10:00:39,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:39,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:39,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:39,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:39,663 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:39,663 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109639663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639663"}]},"ts":"1676109639663"} 2023-02-11 10:00:39,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:39,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:39,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109639657Running coprocessor pre-close hooks at 1676109639657Disabling compacts and flushes for region at 1676109639657Disabling writes for close at 1676109639657Writing region close event to WAL at 1676109639666 (+9 ms)Running coprocessor post-close hooks at 1676109639669 (+3 ms)Closed at 1676109639669 2023-02-11 10:00:39,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109639655Running coprocessor pre-close hooks at 1676109639655Disabling compacts and flushes for region at 1676109639655Disabling writes for close at 1676109639656 (+1 ms)Writing region close event to WAL at 1676109639661 (+5 ms)Running coprocessor post-close hooks at 1676109639669 (+8 ms)Closed at 1676109639669 2023-02-11 10:00:39,670 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=165 2023-02-11 10:00:39,670 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=165, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 178 msec 2023-02-11 10:00:39,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=163 2023-02-11 10:00:39,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=163, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 183 msec 2023-02-11 10:00:39,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:39,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-11 10:00:39,672 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:39,672 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639672"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639672"}]},"ts":"1676109639672"} 2023-02-11 10:00:39,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:39,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:39,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109639660Running coprocessor pre-close hooks at 1676109639660Disabling compacts and flushes for region at 1676109639660Disabling writes for close at 1676109639661 (+1 ms)Writing region close event to WAL at 1676109639666 (+5 ms)Running coprocessor post-close hooks at 1676109639672 (+6 ms)Closed at 1676109639673 (+1 ms) 2023-02-11 10:00:39,673 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 192 msec 2023-02-11 10:00:39,673 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=167 2023-02-11 10:00:39,673 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=167, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 184 msec 2023-02-11 10:00:39,676 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:39,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 193 msec 2023-02-11 10:00:39,676 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109639676"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639676"}]},"ts":"1676109639676"} 2023-02-11 10:00:39,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:39,677 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 195 msec 2023-02-11 10:00:39,677 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:39,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109639677"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109639677"}]},"ts":"1676109639677"} 2023-02-11 10:00:39,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=166 2023-02-11 10:00:39,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=166, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41567,1676109614293 in 192 msec 2023-02-11 10:00:39,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 199 msec 2023-02-11 10:00:39,678 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=162 2023-02-11 10:00:39,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=162, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,41193,1676109614244 in 192 msec 2023-02-11 10:00:39,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=164 2023-02-11 10:00:39,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=164, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,35443,1676109614327 in 189 msec 2023-02-11 10:00:39,679 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 200 msec 2023-02-11 10:00:39,680 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-02-11 10:00:39,680 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 201 msec 2023-02-11 10:00:39,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109639681"}]},"ts":"1676109639681"} 2023-02-11 10:00:39,682 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:39,683 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:39,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 218 msec 2023-02-11 10:00:39,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-11 10:00:39,721 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 161 completed 2023-02-11 10:00:39,726 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,735 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:39,744 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:39,746 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits] 2023-02-11 10:00:39,746 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits] 2023-02-11 10:00:39,747 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits] 2023-02-11 10:00:39,748 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits] 2023-02-11 10:00:39,748 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits] 2023-02-11 10:00:39,749 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits] 2023-02-11 10:00:39,756 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:39,756 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:39,757 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:39,757 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:39,757 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:39,757 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:39,759 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:39,760 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:39,760 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:39,760 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:39,761 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 2023-02-11 10:00:39,762 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:39,766 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/27.seqid 2023-02-11 10:00:39,766 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/27.seqid 2023-02-11 10:00:39,766 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/27.seqid 2023-02-11 10:00:39,767 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/27.seqid 2023-02-11 10:00:39,767 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:39,767 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:39,767 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:39,768 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/27.seqid 2023-02-11 10:00:39,768 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:39,769 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/27.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/27.seqid 2023-02-11 10:00:39,769 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:39,769 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:39,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676109617344 regions 2023-02-11 10:00:39,770 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea 2023-02-11 10:00:39,771 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf] 2023-02-11 10:00:39,774 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:39,775 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:39,776 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:39,777 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:39,779 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:39,780 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:39,781 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:39,783 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:39,784 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:39,785 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:39,787 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:39,788 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:39,788 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea 2023-02-11 10:00:39,791 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,815 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676109617344 from hbase:meta 2023-02-11 10:00:39,817 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676109617344' descriptor. 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676109617344' from region states. 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109639819"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,829 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:39,829 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 10:00:39,829 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676109617344' as deleted. 2023-02-11 10:00:39,830 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676109639830"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:39,831 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676109617344 state from META 2023-02-11 10:00:39,835 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:39,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 in 108 msec 2023-02-11 10:00:39,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-11 10:00:39,989 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 174 completed 2023-02-11 10:00:40,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 ) 2023-02-11 10:00:40,040 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676109617344 as table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 10:00:40,060 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:40,064 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:40,064 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,065 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,067 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,070 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,074 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,079 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,084 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,087 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,100 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,104 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,112 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,115 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,119 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(239): region to add: 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=4f728331505b601d10a557ad43850320 as 4f728331505b601d10a557ad43850320 in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=eb9c4974f7474c0e09c82c0ecda0c585 as eb9c4974f7474c0e09c82c0ecda0c585 in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=1bff2e862183c9fd81aa3c886062178c as 1bff2e862183c9fd81aa3c886062178c in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=59ce432d7225c2fa47a6ee02f27bbfdc as 59ce432d7225c2fa47a6ee02f27bbfdc in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,123 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=b714c90af14d8acfa1b06ca5768827da as b714c90af14d8acfa1b06ca5768827da in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,124 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(585): clone region=86a1312865cab1cc55298ab2c74451da as 86a1312865cab1cc55298ab2c74451da in snapshot snaptb1-1676109617344 2023-02-11 10:00:40,125 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,126 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,126 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,131 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,138 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,139 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676109617344', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:40,172 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,172 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7de4e72b64f140619c53542c482912d0 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,192 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,192 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7272f33cb3994cf39cac4f156ba82a23 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,197 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7558e76cf7794bd9821d73896593011f from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,200 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink fa1f306b33c8410e8fc7746a0d41671f from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,200 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,200 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 843d3d9439ae414e8d30988488444e12 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,200 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,200 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 21dc81187d984457854aefc08e61f8d8 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,202 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:40,202 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,202 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,202 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:40,202 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,202 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,202 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109640202Disabling compacts and flushes for region at 1676109640202Disabling writes for close at 1676109640202Writing region close event to WAL at 1676109640202Closed at 1676109640202 2023-02-11 10:00:40,206 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:40,206 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:40,206 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:40,206 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:40,206 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:40,206 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:40,206 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109640206Disabling compacts and flushes for region at 1676109640206Disabling writes for close at 1676109640206Writing region close event to WAL at 1676109640206Closed at 1676109640206 2023-02-11 10:00:40,212 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3719691d68774d919d12001277aeabf7 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,213 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f700cba99b9d45a6939369b56232ede4 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,213 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,213 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 98464fb3faf34d1ba2a19a37143ccc6f from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,221 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:40,221 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:40,221 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:40,221 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:40,221 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:40,221 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:40,221 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109640221Disabling compacts and flushes for region at 1676109640221Disabling writes for close at 1676109640221Writing region close event to WAL at 1676109640221Closed at 1676109640221 2023-02-11 10:00:40,224 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ca87f195a3104b539048984ba5a3a37c from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,226 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:40,226 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:40,226 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:40,226 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:40,226 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:40,226 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:40,226 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109640226Disabling compacts and flushes for region at 1676109640226Disabling writes for close at 1676109640226Writing region close event to WAL at 1676109640226Closed at 1676109640226 2023-02-11 10:00:40,228 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:40,228 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,228 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,228 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:40,228 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,229 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,229 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109640228Disabling compacts and flushes for region at 1676109640228Disabling writes for close at 1676109640228Writing region close event to WAL at 1676109640229 (+1 ms)Closed at 1676109640229 2023-02-11 10:00:40,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 10:00:40,590 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,590 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 509bed9065f84d0ba31ce69a3a3f4bd0 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,593 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c8e02b793157422b98df2d998ef0b4c8 from cloned region in snapshot snaptb1-1676109617344 to table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:40,595 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:40,595 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,595 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,595 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:40,595 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,595 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,595 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109640595Disabling compacts and flushes for region at 1676109640595Disabling writes for close at 1676109640595Writing region close event to WAL at 1676109640595Closed at 1676109640595 2023-02-11 10:00:40,596 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 2023-02-11 10:00:40,596 INFO [PEWorker-4] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676109617344 on table=testRestoreSnapshot-1676109617344 completed! 2023-02-11 10:00:40,601 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,602 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,602 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,602 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,602 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,602 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109640601"}]},"ts":"1676109640601"} 2023-02-11 10:00:40,604 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:40,605 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109640605"}]},"ts":"1676109640605"} 2023-02-11 10:00:40,606 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLING in hbase:meta 2023-02-11 10:00:40,611 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:40,611 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:40,611 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:40,611 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:40,611 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:40,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN}] 2023-02-11 10:00:40,614 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN 2023-02-11 10:00:40,614 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN 2023-02-11 10:00:40,614 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN 2023-02-11 10:00:40,614 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN 2023-02-11 10:00:40,616 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:40,616 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN 2023-02-11 10:00:40,616 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:40,616 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:40,616 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:40,617 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN 2023-02-11 10:00:40,617 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:40,617 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:40,767 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:40,771 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:40,771 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:40,772 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640771"}]},"ts":"1676109640771"} 2023-02-11 10:00:40,771 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:40,771 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:40,772 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109640771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640771"}]},"ts":"1676109640771"} 2023-02-11 10:00:40,771 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:40,772 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640771"}]},"ts":"1676109640771"} 2023-02-11 10:00:40,772 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640771"}]},"ts":"1676109640771"} 2023-02-11 10:00:40,772 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640771"}]},"ts":"1676109640771"} 2023-02-11 10:00:40,774 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=179, state=RUNNABLE; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:40,779 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=181, state=RUNNABLE; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:40,779 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=178, state=RUNNABLE; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:40,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=177, state=RUNNABLE; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:40,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=180, state=RUNNABLE; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:40,783 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:40,783 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109640783"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109640783"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109640783"}]},"ts":"1676109640783"} 2023-02-11 10:00:40,785 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=176, state=RUNNABLE; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:40,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 10:00:40,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:40,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,938 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,938 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,938 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:40,939 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:40,939 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf 2023-02-11 10:00:40,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,939 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf 2023-02-11 10:00:40,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,939 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f728331505b601d10a557ad43850320 columnFamilyName cf 2023-02-11 10:00:40,939 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eb9c4974f7474c0e09c82c0ecda0c585 columnFamilyName cf 2023-02-11 10:00:40,940 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,940 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:40,941 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf 2023-02-11 10:00:40,941 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 86a1312865cab1cc55298ab2c74451da columnFamilyName cf 2023-02-11 10:00:40,950 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:40,951 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:40,952 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:40,959 DEBUG [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:40,959 INFO [StoreOpener-4f728331505b601d10a557ad43850320-1] regionserver.HStore(310): Store=4f728331505b601d10a557ad43850320/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:40,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,962 DEBUG [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:40,962 INFO [StoreOpener-eb9c4974f7474c0e09c82c0ecda0c585-1] regionserver.HStore(310): Store=eb9c4974f7474c0e09c82c0ecda0c585/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:40,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f728331505b601d10a557ad43850320 2023-02-11 10:00:40,968 DEBUG [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:40,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:40,969 INFO [StoreOpener-86a1312865cab1cc55298ab2c74451da-1] regionserver.HStore(310): Store=86a1312865cab1cc55298ab2c74451da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:40,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:40,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f728331505b601d10a557ad43850320; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73887228, jitterRate=0.10100549459457397}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:40,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f728331505b601d10a557ad43850320: Running coprocessor pre-open hook at 1676109640936Writing region info on filesystem at 1676109640936Initializing all the Stores at 1676109640937 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640937Cleaning up temporary data from old regions at 1676109640962 (+25 ms)Cleaning up detritus from prior splits at 1676109640964 (+2 ms)Running coprocessor post-open hooks at 1676109640970 (+6 ms)Region opened successfully at 1676109640970 2023-02-11 10:00:40,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320., pid=185, masterSystemTime=1676109640932 2023-02-11 10:00:40,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:40,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:40,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:40,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:40,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:40,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:40,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:40,975 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:40,975 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640974"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109640974"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109640974"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109640974"}]},"ts":"1676109640974"} 2023-02-11 10:00:40,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:40,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:40,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eb9c4974f7474c0e09c82c0ecda0c585; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70097028, jitterRate=0.04452711343765259}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:40,976 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:40,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eb9c4974f7474c0e09c82c0ecda0c585: Running coprocessor pre-open hook at 1676109640937Writing region info on filesystem at 1676109640937Initializing all the Stores at 1676109640937Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640937Cleaning up temporary data from old regions at 1676109640965 (+28 ms)Cleaning up detritus from prior splits at 1676109640966 (+1 ms)Running coprocessor post-open hooks at 1676109640976 (+10 ms)Region opened successfully at 1676109640976 2023-02-11 10:00:40,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 86a1312865cab1cc55298ab2c74451da; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59599764, jitterRate=-0.11189430952072144}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:40,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 86a1312865cab1cc55298ab2c74451da: Running coprocessor pre-open hook at 1676109640939Writing region info on filesystem at 1676109640939Initializing all the Stores at 1676109640939Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640939Cleaning up temporary data from old regions at 1676109640970 (+31 ms)Cleaning up detritus from prior splits at 1676109640971 (+1 ms)Running coprocessor post-open hooks at 1676109640976 (+5 ms)Region opened successfully at 1676109640976 2023-02-11 10:00:40,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585., pid=187, masterSystemTime=1676109640933 2023-02-11 10:00:40,977 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:40,977 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf 2023-02-11 10:00:40,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da., pid=186, masterSystemTime=1676109640932 2023-02-11 10:00:40,979 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=177 2023-02-11 10:00:40,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:40,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:40,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:40,979 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:40,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:40,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,980 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=177, state=SUCCESS; OpenRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 195 msec 2023-02-11 10:00:40,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:40,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:40,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:40,980 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109640979"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109640979"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109640979"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109640979"}]},"ts":"1676109640979"} 2023-02-11 10:00:40,980 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 59ce432d7225c2fa47a6ee02f27bbfdc columnFamilyName cf 2023-02-11 10:00:40,980 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:40,980 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, ASSIGN in 367 msec 2023-02-11 10:00:40,980 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109640980"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109640980"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109640980"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109640980"}]},"ts":"1676109640980"} 2023-02-11 10:00:40,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676109617344 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:40,981 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:40,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:40,982 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:40,982 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf 2023-02-11 10:00:40,983 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b714c90af14d8acfa1b06ca5768827da columnFamilyName cf 2023-02-11 10:00:40,984 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=176 2023-02-11 10:00:40,984 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=176, state=SUCCESS; OpenRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 196 msec 2023-02-11 10:00:40,984 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:40,986 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=180 2023-02-11 10:00:40,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, ASSIGN in 372 msec 2023-02-11 10:00:40,986 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=180, state=SUCCESS; OpenRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 202 msec 2023-02-11 10:00:40,986 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:40,986 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf 2023-02-11 10:00:40,986 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bff2e862183c9fd81aa3c886062178c columnFamilyName cf 2023-02-11 10:00:40,988 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, ASSIGN in 374 msec 2023-02-11 10:00:40,996 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:40,997 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:41,002 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:41,002 DEBUG [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:41,005 INFO [StoreOpener-59ce432d7225c2fa47a6ee02f27bbfdc-1] regionserver.HStore(310): Store=59ce432d7225c2fa47a6ee02f27bbfdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:41,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:41,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:41,008 DEBUG [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 2023-02-11 10:00:41,008 INFO [StoreOpener-b714c90af14d8acfa1b06ca5768827da-1] regionserver.HStore(310): Store=b714c90af14d8acfa1b06ca5768827da/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:41,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:41,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:41,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:41,012 DEBUG [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:41,012 INFO [StoreOpener-1bff2e862183c9fd81aa3c886062178c-1] regionserver.HStore(310): Store=1bff2e862183c9fd81aa3c886062178c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:41,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:41,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:41,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:41,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 59ce432d7225c2fa47a6ee02f27bbfdc; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71660250, jitterRate=0.06782093644142151}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:41,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Running coprocessor pre-open hook at 1676109640974Writing region info on filesystem at 1676109640974Initializing all the Stores at 1676109640975 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640975Cleaning up temporary data from old regions at 1676109641006 (+31 ms)Cleaning up detritus from prior splits at 1676109641009 (+3 ms)Running coprocessor post-open hooks at 1676109641016 (+7 ms)Region opened successfully at 1676109641016 2023-02-11 10:00:41,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:41,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc., pid=184, masterSystemTime=1676109640932 2023-02-11 10:00:41,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:41,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:41,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:41,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:41,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b714c90af14d8acfa1b06ca5768827da; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65139633, jitterRate=-0.029343828558921814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:41,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b714c90af14d8acfa1b06ca5768827da: Running coprocessor pre-open hook at 1676109640980Writing region info on filesystem at 1676109640980Initializing all the Stores at 1676109640981 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640981Cleaning up temporary data from old regions at 1676109641012 (+31 ms)Cleaning up detritus from prior splits at 1676109641012Running coprocessor post-open hooks at 1676109641018 (+6 ms)Region opened successfully at 1676109641018 2023-02-11 10:00:41,019 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:41,019 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109641019"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109641019"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109641019"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109641019"}]},"ts":"1676109641019"} 2023-02-11 10:00:41,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da., pid=183, masterSystemTime=1676109640933 2023-02-11 10:00:41,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-11 10:00:41,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1bff2e862183c9fd81aa3c886062178c; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59934241, jitterRate=-0.10691021382808685}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:41,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1bff2e862183c9fd81aa3c886062178c: Running coprocessor pre-open hook at 1676109640981Writing region info on filesystem at 1676109640981Initializing all the Stores at 1676109640983 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109640983Cleaning up temporary data from old regions at 1676109641015 (+32 ms)Cleaning up detritus from prior splits at 1676109641016 (+1 ms)Running coprocessor post-open hooks at 1676109641025 (+9 ms)Region opened successfully at 1676109641025 2023-02-11 10:00:41,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:41,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c., pid=182, masterSystemTime=1676109640932 2023-02-11 10:00:41,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:41,026 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:41,026 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109641026"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109641026"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109641026"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109641026"}]},"ts":"1676109641026"} 2023-02-11 10:00:41,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:41,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:41,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=178 2023-02-11 10:00:41,027 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:41,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=178, state=SUCCESS; OpenRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41567,1676109614293 in 246 msec 2023-02-11 10:00:41,028 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109641027"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109641027"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109641027"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109641027"}]},"ts":"1676109641027"} 2023-02-11 10:00:41,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, ASSIGN in 415 msec 2023-02-11 10:00:41,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=181 2023-02-11 10:00:41,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=181, state=SUCCESS; OpenRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 253 msec 2023-02-11 10:00:41,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, ASSIGN in 418 msec 2023-02-11 10:00:41,031 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=179 2023-02-11 10:00:41,032 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=179, state=SUCCESS; OpenRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 256 msec 2023-02-11 10:00:41,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=179, resume processing ppid=175 2023-02-11 10:00:41,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, ASSIGN in 420 msec 2023-02-11 10:00:41,034 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109641034"}]},"ts":"1676109641034"} 2023-02-11 10:00:41,035 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=ENABLED in hbase:meta 2023-02-11 10:00:41,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676109617344 snapshot=name: "snaptb1-1676109617344" table: "testRestoreSnapshot-1676109617344" creation_time: 1676109628646 type: DISABLED version: 2 ttl: 0 ) in 1.0070 sec 2023-02-11 10:00:41,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-11 10:00:41,551 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676109617344, procId: 175 completed 2023-02-11 10:00:41,552 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(139): Connect 0x1913e027 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:41,560 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ef28d5f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:41,563 DEBUG [hconnection-0x3361d618-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:41,565 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46138, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:41,569 DEBUG [hconnection-0x3361d618-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:41,572 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52860, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:41,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,818 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:41,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,206 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:42,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,717 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:42,748 DEBUG [hconnection-0x3361d618-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:42,753 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:42,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:42,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,348 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,718 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:43,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:43,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:44,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,157 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:45,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,328 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29616110 to 127.0.0.1:64863 2023-02-11 10:00:45,329 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:45,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,345 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676109617344' 2023-02-11 10:00:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:46,614 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676109617344 2023-02-11 10:00:46,614 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676109617344 2023-02-11 10:00:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:46,619 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109646619"}]},"ts":"1676109646619"} 2023-02-11 10:00:46,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 10:00:46,620 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLING in hbase:meta 2023-02-11 10:00:46,622 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676109617344 to state=DISABLING 2023-02-11 10:00:46,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN}] 2023-02-11 10:00:46,624 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN 2023-02-11 10:00:46,626 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN 2023-02-11 10:00:46,626 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN 2023-02-11 10:00:46,627 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN 2023-02-11 10:00:46,627 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN 2023-02-11 10:00:46,627 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:46,627 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:46,627 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:46,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109646627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646627"}]},"ts":"1676109646627"} 2023-02-11 10:00:46,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646627"}]},"ts":"1676109646627"} 2023-02-11 10:00:46,628 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:46,628 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646628"}]},"ts":"1676109646628"} 2023-02-11 10:00:46,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646627"}]},"ts":"1676109646627"} 2023-02-11 10:00:46,627 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:46,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646627"}]},"ts":"1676109646627"} 2023-02-11 10:00:46,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=194, state=RUNNABLE; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:46,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=191, state=RUNNABLE; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:46,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=192, state=RUNNABLE; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:46,643 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=190, state=RUNNABLE; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:46,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:46,646 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN 2023-02-11 10:00:46,649 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:46,649 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109646649"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109646649"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109646649"}]},"ts":"1676109646649"} 2023-02-11 10:00:46,660 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=189, state=RUNNABLE; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:46,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:46,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eb9c4974f7474c0e09c82c0ecda0c585, disabling compactions & flushes 2023-02-11 10:00:46,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:46,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:46,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. after waiting 0 ms 2023-02-11 10:00:46,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:46,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f728331505b601d10a557ad43850320 2023-02-11 10:00:46,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:46,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f728331505b601d10a557ad43850320, disabling compactions & flushes 2023-02-11 10:00:46,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:46,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. after waiting 0 ms 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1bff2e862183c9fd81aa3c886062178c, disabling compactions & flushes 2023-02-11 10:00:46,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. after waiting 0 ms 2023-02-11 10:00:46,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:46,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320. 2023-02-11 10:00:46,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f728331505b601d10a557ad43850320: Waiting for close lock at 1676109646795Running coprocessor pre-close hooks at 1676109646795Disabling compacts and flushes for region at 1676109646795Disabling writes for close at 1676109646798 (+3 ms)Writing region close event to WAL at 1676109646844 (+46 ms)Running coprocessor post-close hooks at 1676109646856 (+12 ms)Closed at 1676109646859 (+3 ms) 2023-02-11 10:00:46,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c. 2023-02-11 10:00:46,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1bff2e862183c9fd81aa3c886062178c: Waiting for close lock at 1676109646796Running coprocessor pre-close hooks at 1676109646796Disabling compacts and flushes for region at 1676109646796Disabling writes for close at 1676109646798 (+2 ms)Writing region close event to WAL at 1676109646828 (+30 ms)Running coprocessor post-close hooks at 1676109646859 (+31 ms)Closed at 1676109646860 (+1 ms) 2023-02-11 10:00:46,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f728331505b601d10a557ad43850320 2023-02-11 10:00:46,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:46,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585. 2023-02-11 10:00:46,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eb9c4974f7474c0e09c82c0ecda0c585: Waiting for close lock at 1676109646791Running coprocessor pre-close hooks at 1676109646791Disabling compacts and flushes for region at 1676109646791Disabling writes for close at 1676109646792 (+1 ms)Writing region close event to WAL at 1676109646844 (+52 ms)Running coprocessor post-close hooks at 1676109646860 (+16 ms)Closed at 1676109646863 (+3 ms) 2023-02-11 10:00:46,863 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=4f728331505b601d10a557ad43850320, regionState=CLOSED 2023-02-11 10:00:46,864 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646863"}]},"ts":"1676109646863"} 2023-02-11 10:00:46,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:46,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:46,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 59ce432d7225c2fa47a6ee02f27bbfdc, disabling compactions & flushes 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 86a1312865cab1cc55298ab2c74451da, disabling compactions & flushes 2023-02-11 10:00:46,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. after waiting 0 ms 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:46,865 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. after waiting 0 ms 2023-02-11 10:00:46,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:46,867 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=1bff2e862183c9fd81aa3c886062178c, regionState=CLOSED 2023-02-11 10:00:46,867 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646867"}]},"ts":"1676109646867"} 2023-02-11 10:00:46,868 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=eb9c4974f7474c0e09c82c0ecda0c585, regionState=CLOSED 2023-02-11 10:00:46,868 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109646867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646867"}]},"ts":"1676109646867"} 2023-02-11 10:00:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 10:00:46,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:46,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:46,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=190 2023-02-11 10:00:46,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=190, state=SUCCESS; CloseRegionProcedure 4f728331505b601d10a557ad43850320, server=jenkins-hbase4.apache.org,41567,1676109614293 in 228 msec 2023-02-11 10:00:46,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=192 2023-02-11 10:00:46,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=4f728331505b601d10a557ad43850320, UNASSIGN in 251 msec 2023-02-11 10:00:46,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=192, state=SUCCESS; CloseRegionProcedure 1bff2e862183c9fd81aa3c886062178c, server=jenkins-hbase4.apache.org,41193,1676109614244 in 230 msec 2023-02-11 10:00:46,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=189 2023-02-11 10:00:46,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=189, state=SUCCESS; CloseRegionProcedure eb9c4974f7474c0e09c82c0ecda0c585, server=jenkins-hbase4.apache.org,35443,1676109614327 in 209 msec 2023-02-11 10:00:46,879 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=1bff2e862183c9fd81aa3c886062178c, UNASSIGN in 254 msec 2023-02-11 10:00:46,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=eb9c4974f7474c0e09c82c0ecda0c585, UNASSIGN in 254 msec 2023-02-11 10:00:46,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b714c90af14d8acfa1b06ca5768827da, disabling compactions & flushes 2023-02-11 10:00:46,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:46,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:46,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. after waiting 0 ms 2023-02-11 10:00:46,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:46,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da. 2023-02-11 10:00:46,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b714c90af14d8acfa1b06ca5768827da: Waiting for close lock at 1676109646871Running coprocessor pre-close hooks at 1676109646871Disabling compacts and flushes for region at 1676109646871Disabling writes for close at 1676109646881 (+10 ms)Writing region close event to WAL at 1676109646922 (+41 ms)Running coprocessor post-close hooks at 1676109646930 (+8 ms)Closed at 1676109646939 (+9 ms) 2023-02-11 10:00:46,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:46,942 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=b714c90af14d8acfa1b06ca5768827da, regionState=CLOSED 2023-02-11 10:00:46,942 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676109646942"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646942"}]},"ts":"1676109646942"} 2023-02-11 10:00:46,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=194 2023-02-11 10:00:46,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=194, state=SUCCESS; CloseRegionProcedure b714c90af14d8acfa1b06ca5768827da, server=jenkins-hbase4.apache.org,35443,1676109614327 in 307 msec 2023-02-11 10:00:46,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=b714c90af14d8acfa1b06ca5768827da, UNASSIGN in 323 msec 2023-02-11 10:00:46,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-11 10:00:46,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc. 2023-02-11 10:00:46,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 59ce432d7225c2fa47a6ee02f27bbfdc: Waiting for close lock at 1676109646863Running coprocessor pre-close hooks at 1676109646863Disabling compacts and flushes for region at 1676109646863Disabling writes for close at 1676109646865 (+2 ms)Writing region close event to WAL at 1676109646928 (+63 ms)Running coprocessor post-close hooks at 1676109646938 (+10 ms)Closed at 1676109646963 (+25 ms) 2023-02-11 10:00:46,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da. 2023-02-11 10:00:46,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 86a1312865cab1cc55298ab2c74451da: Waiting for close lock at 1676109646864Running coprocessor pre-close hooks at 1676109646864Disabling compacts and flushes for region at 1676109646864Disabling writes for close at 1676109646865 (+1 ms)Writing region close event to WAL at 1676109646928 (+63 ms)Running coprocessor post-close hooks at 1676109646956 (+28 ms)Closed at 1676109646963 (+7 ms) 2023-02-11 10:00:46,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:46,967 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=59ce432d7225c2fa47a6ee02f27bbfdc, regionState=CLOSED 2023-02-11 10:00:46,967 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646967"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646967"}]},"ts":"1676109646967"} 2023-02-11 10:00:46,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:46,968 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=86a1312865cab1cc55298ab2c74451da, regionState=CLOSED 2023-02-11 10:00:46,969 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676109646968"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109646968"}]},"ts":"1676109646968"} 2023-02-11 10:00:46,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=191 2023-02-11 10:00:46,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=191, state=SUCCESS; CloseRegionProcedure 59ce432d7225c2fa47a6ee02f27bbfdc, server=jenkins-hbase4.apache.org,41567,1676109614293 in 332 msec 2023-02-11 10:00:46,975 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-02-11 10:00:46,975 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure 86a1312865cab1cc55298ab2c74451da, server=jenkins-hbase4.apache.org,41193,1676109614244 in 327 msec 2023-02-11 10:00:46,976 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=59ce432d7225c2fa47a6ee02f27bbfdc, UNASSIGN in 351 msec 2023-02-11 10:00:46,978 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=193, resume processing ppid=188 2023-02-11 10:00:46,978 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676109617344, region=86a1312865cab1cc55298ab2c74451da, UNASSIGN in 352 msec 2023-02-11 10:00:46,979 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109646979"}]},"ts":"1676109646979"} 2023-02-11 10:00:46,980 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676109617344, state=DISABLED in hbase:meta 2023-02-11 10:00:46,982 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676109617344 to state=DISABLED 2023-02-11 10:00:46,985 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676109617344 in 369 msec 2023-02-11 10:00:47,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-11 10:00:47,372 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 188 completed 2023-02-11 10:00:47,373 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,377 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,378 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:47,390 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:47,393 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits] 2023-02-11 10:00:47,393 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits] 2023-02-11 10:00:47,393 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits] 2023-02-11 10:00:47,394 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits] 2023-02-11 10:00:47,394 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits] 2023-02-11 10:00:47,395 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits] 2023-02-11 10:00:47,403 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639755 currentMtime: 1676109640212 2023-02-11 10:00:47,405 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639755 currentMtime: 1676109640185 2023-02-11 10:00:47,405 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639755 currentMtime: 1676109640197 2023-02-11 10:00:47,405 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639755 currentMtime: 1676109640592 2023-02-11 10:00:47,405 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8.1676109647394. 2023-02-11 10:00:47,406 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639755 currentMtime: 1676109640212 2023-02-11 10:00:47,406 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0.1676109647393. 2023-02-11 10:00:47,407 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12.1676109647393. 2023-02-11 10:00:47,407 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0.1676109647394. 2023-02-11 10:00:47,407 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23.1676109647393. 2023-02-11 10:00:47,407 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639756 currentMtime: 1676109640220 2023-02-11 10:00:47,408 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-21dc81187d984457854aefc08e61f8d8 2023-02-11 10:00:47,408 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f.1676109647395. 2023-02-11 10:00:47,408 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-7de4e72b64f140619c53542c482912d0 2023-02-11 10:00:47,409 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-509bed9065f84d0ba31ce69a3a3f4bd0 2023-02-11 10:00:47,409 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-843d3d9439ae414e8d30988488444e12 2023-02-11 10:00:47,410 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7272f33cb3994cf39cac4f156ba82a23 2023-02-11 10:00:47,411 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639758 currentMtime: 1676109640219 2023-02-11 10:00:47,411 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639758 currentMtime: 1676109640595 2023-02-11 10:00:47,411 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639757 currentMtime: 1676109640225 2023-02-11 10:00:47,411 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-98464fb3faf34d1ba2a19a37143ccc6f 2023-02-11 10:00:47,412 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639757 currentMtime: 1676109640205 2023-02-11 10:00:47,412 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7.1676109647394. 2023-02-11 10:00:47,412 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4.1676109647393. 2023-02-11 10:00:47,412 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8.1676109647394. 2023-02-11 10:00:47,412 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f.1676109647393. 2023-02-11 10:00:47,413 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639758 currentMtime: 1676109640200 2023-02-11 10:00:47,414 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/cf/testRestoreSnapshot-1676109617344=1bff2e862183c9fd81aa3c886062178c-f700cba99b9d45a6939369b56232ede4 2023-02-11 10:00:47,414 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f.1676109647393. 2023-02-11 10:00:47,415 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/cf/testRestoreSnapshot-1676109617344=59ce432d7225c2fa47a6ee02f27bbfdc-fa1f306b33c8410e8fc7746a0d41671f 2023-02-11 10:00:47,415 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/cf/testRestoreSnapshot-1676109617344=b714c90af14d8acfa1b06ca5768827da-3719691d68774d919d12001277aeabf7 2023-02-11 10:00:47,416 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/cf/testRestoreSnapshot-1676109617344=eb9c4974f7474c0e09c82c0ecda0c585-c8e02b793157422b98df2d998ef0b4c8 2023-02-11 10:00:47,416 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/cf/testRestoreSnapshot-1676109617344=4f728331505b601d10a557ad43850320-7558e76cf7794bd9821d73896593011f 2023-02-11 10:00:47,417 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639759 currentMtime: 1676109640228 2023-02-11 10:00:47,418 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c.1676109647395. 2023-02-11 10:00:47,421 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/cf/testRestoreSnapshot-1676109617344=86a1312865cab1cc55298ab2c74451da-ca87f195a3104b539048984ba5a3a37c 2023-02-11 10:00:47,422 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da/recovered.edits/18.seqid 2023-02-11 10:00:47,422 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c/recovered.edits/18.seqid 2023-02-11 10:00:47,423 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320/recovered.edits/18.seqid 2023-02-11 10:00:47,423 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/b714c90af14d8acfa1b06ca5768827da 2023-02-11 10:00:47,423 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/1bff2e862183c9fd81aa3c886062178c 2023-02-11 10:00:47,424 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/4f728331505b601d10a557ad43850320 2023-02-11 10:00:47,424 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc/recovered.edits/18.seqid 2023-02-11 10:00:47,424 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585/recovered.edits/18.seqid 2023-02-11 10:00:47,425 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/59ce432d7225c2fa47a6ee02f27bbfdc 2023-02-11 10:00:47,425 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/eb9c4974f7474c0e09c82c0ecda0c585 2023-02-11 10:00:47,426 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da/recovered.edits/18.seqid 2023-02-11 10:00:47,427 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testRestoreSnapshot-1676109617344/86a1312865cab1cc55298ab2c74451da 2023-02-11 10:00:47,428 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676109617344 regions 2023-02-11 10:00:47,428 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea 2023-02-11 10:00:47,429 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf] 2023-02-11 10:00:47,432 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639773 currentMtime: 1676109640067 2023-02-11 10:00:47,433 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d.1676109647429. 2023-02-11 10:00:47,434 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c202302116447963f5a394148b7b18a4e2824996d 2023-02-11 10:00:47,436 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639774 currentMtime: 1676109640070 2023-02-11 10:00:47,437 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7.1676109647429. 2023-02-11 10:00:47,438 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-a87ff679a2f3e71d9181a67b7542122c20230211f5578686847f4b4fa3d1f31627936ec7 2023-02-11 10:00:47,440 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639775 currentMtime: 1676109640074 2023-02-11 10:00:47,441 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861.1676109647429. 2023-02-11 10:00:47,444 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110af5fdbc068648e8bff2584218a22861 2023-02-11 10:00:47,446 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639777 currentMtime: 1676109640079 2023-02-11 10:00:47,446 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3.1676109647429. 2023-02-11 10:00:47,447 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c4ca4238a0b923820dcc509a6f75849b202302110d1ff5141069406f964ea62d3709d7b3 2023-02-11 10:00:47,448 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639778 currentMtime: 1676109640083 2023-02-11 10:00:47,449 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac.1676109647429. 2023-02-11 10:00:47,450 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211af5c4aabe2a84ca1a9500853bbe916ac 2023-02-11 10:00:47,451 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639779 currentMtime: 1676109640087 2023-02-11 10:00:47,451 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38.1676109647429. 2023-02-11 10:00:47,452 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-c81e728d9d4c2f636f067f89cc14862c20230211e94500428cf7496586f9c085dad3ef38 2023-02-11 10:00:47,453 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639781 currentMtime: 1676109640100 2023-02-11 10:00:47,454 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425.1676109647429. 2023-02-11 10:00:47,455 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211577da49a5966437abaa7b6fa98a24425 2023-02-11 10:00:47,457 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639782 currentMtime: 1676109640104 2023-02-11 10:00:47,457 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06.1676109647429. 2023-02-11 10:00:47,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-d41d8cd98f00b204e9800998ecf8427e20230211a180941eeacb4c95b4aa58ee1082ac06 2023-02-11 10:00:47,459 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639783 currentMtime: 1676109640111 2023-02-11 10:00:47,459 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241.1676109647429. 2023-02-11 10:00:47,460 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d5202302115109af130df240b2a33f44de0c3ee241 2023-02-11 10:00:47,462 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639785 currentMtime: 1676109640114 2023-02-11 10:00:47,462 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad.1676109647429. 2023-02-11 10:00:47,464 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-e4da3b7fbbce2345d7772b0674a318d520230211f61ab42321d74c80be61c939ffa7cbad 2023-02-11 10:00:47,466 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639786 currentMtime: 1676109640119 2023-02-11 10:00:47,467 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa.1676109647429. 2023-02-11 10:00:47,468 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf3202302119f5c44a99fcd48eb99bda20ba50ed4fa 2023-02-11 10:00:47,470 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823. archiveLen: 0 currentLen: 0 archiveMtime: 1676109639787 currentMtime: 1676109640122 2023-02-11 10:00:47,470 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823.1676109647429. 2023-02-11 10:00:47,471 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea/cf/testRestoreSnapshot-1676109617344=ddb1336944af1bf0427d1343326835ea-eccbc87e4b5ce2fe28308fd9f2a7baf320230211bf8d1f691685490c8acd47d6b2c50823 2023-02-11 10:00:47,471 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testRestoreSnapshot-1676109617344/ddb1336944af1bf0427d1343326835ea 2023-02-11 10:00:47,474 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,478 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676109617344 from hbase:meta 2023-02-11 10:00:47,483 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676109617344' descriptor. 2023-02-11 10:00:47,484 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,484 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676109617344' from region states. 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,485 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109647485"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,493 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:47,493 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => eb9c4974f7474c0e09c82c0ecda0c585, NAME => 'testRestoreSnapshot-1676109617344,,1676109617373.eb9c4974f7474c0e09c82c0ecda0c585.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 4f728331505b601d10a557ad43850320, NAME => 'testRestoreSnapshot-1676109617344,1,1676109617373.4f728331505b601d10a557ad43850320.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 59ce432d7225c2fa47a6ee02f27bbfdc, NAME => 'testRestoreSnapshot-1676109617344,2,1676109617373.59ce432d7225c2fa47a6ee02f27bbfdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 1bff2e862183c9fd81aa3c886062178c, NAME => 'testRestoreSnapshot-1676109617344,3,1676109617373.1bff2e862183c9fd81aa3c886062178c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 86a1312865cab1cc55298ab2c74451da, NAME => 'testRestoreSnapshot-1676109617344,4,1676109617373.86a1312865cab1cc55298ab2c74451da.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b714c90af14d8acfa1b06ca5768827da, NAME => 'testRestoreSnapshot-1676109617344,5,1676109617373.b714c90af14d8acfa1b06ca5768827da.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 10:00:47,493 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676109617344' as deleted. 2023-02-11 10:00:47,493 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676109617344","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676109647493"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:47,502 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676109617344 state from META 2023-02-11 10:00:47,508 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 2023-02-11 10:00:47,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676109617344 in 135 msec 2023-02-11 10:00:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-11 10:00:47,631 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676109617344, procId: 201 completed 2023-02-11 10:00:47,640 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676109617344" 2023-02-11 10:00:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676109617344 2023-02-11 10:00:47,646 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676109617344" 2023-02-11 10:00:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676109617344 2023-02-11 10:00:47,649 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676109617344" 2023-02-11 10:00:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676109617344 2023-02-11 10:00:47,698 INFO [Listener at localhost/42305] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=407 (was 359) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:64863@0x1913e027-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:36662 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:36502 [Waiting for operation #536] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:46404 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:64863@0x1913e027-SendThread(127.0.0.1:64863) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:52366 [Waiting for operation #464] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:52376 [Waiting for operation #469] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3361d618-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:54484 [Waiting for operation #510] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:64863@0x1913e027 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/578259633.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=774 (was 691) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=335 (was 193) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2331 (was 3285) 2023-02-11 10:00:47,730 INFO [Listener at localhost/42305] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=407, OpenFileDescriptor=774, MaxFileDescriptor=60000, SystemLoadAverage=335, ProcessCount=167, AvailableMemoryMB=2329 2023-02-11 10:00:47,734 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-11 10:00:47,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:47,738 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 10:00:47,738 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676109647730" procId is: 202 2023-02-11 10:00:47,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 10:00:47,741 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:47,747 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 empty. 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e empty. 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 empty. 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca empty. 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:47,748 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d empty. 2023-02-11 10:00:47,749 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 empty. 2023-02-11 10:00:47,749 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:47,749 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:47,749 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:47,750 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:47,750 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:47,750 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676109647730 regions 2023-02-11 10:00:47,800 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/.tabledesc/.tableinfo.0000000001 2023-02-11 10:00:47,805 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(7671): creating {ENCODED => 2dba5f8399997a147e997c4d38161234, NAME => 'testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,806 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(7671): creating {ENCODED => c792e2d4f0faf7086352849794012c79, NAME => 'testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,806 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(7671): creating {ENCODED => 4f895b4386bc67b999d38d2505ce37ca, NAME => 'testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1603): Closing 4f895b4386bc67b999d38d2505ce37ca, disabling compactions & flushes 2023-02-11 10:00:47,840 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. after waiting 0 ms 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:47,840 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:47,840 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1557): Region close journal for 4f895b4386bc67b999d38d2505ce37ca: Waiting for close lock at 1676109647840Disabling compacts and flushes for region at 1676109647840Disabling writes for close at 1676109647840Writing region close event to WAL at 1676109647840Closed at 1676109647840 2023-02-11 10:00:47,841 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(7671): creating {ENCODED => 613f74e8c853d032f33c9b749dc5b78d, NAME => 'testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,861 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:47,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1603): Closing 613f74e8c853d032f33c9b749dc5b78d, disabling compactions & flushes 2023-02-11 10:00:47,862 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:47,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:47,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. after waiting 0 ms 2023-02-11 10:00:47,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:47,862 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:47,862 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1557): Region close journal for 613f74e8c853d032f33c9b749dc5b78d: Waiting for close lock at 1676109647862Disabling compacts and flushes for region at 1676109647862Disabling writes for close at 1676109647862Writing region close event to WAL at 1676109647862Closed at 1676109647862 2023-02-11 10:00:47,862 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(7671): creating {ENCODED => 5b412f238605ea2c2123a33e84ce883e, NAME => 'testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1603): Closing 5b412f238605ea2c2123a33e84ce883e, disabling compactions & flushes 2023-02-11 10:00:47,879 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. after waiting 0 ms 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:47,879 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:47,879 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1557): Region close journal for 5b412f238605ea2c2123a33e84ce883e: Waiting for close lock at 1676109647879Disabling compacts and flushes for region at 1676109647879Disabling writes for close at 1676109647879Writing region close event to WAL at 1676109647879Closed at 1676109647879 2023-02-11 10:00:47,880 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(7671): creating {ENCODED => 1864ff4284b6d01e4d7d9b8e96f54a40, NAME => 'testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676109647730', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1603): Closing 1864ff4284b6d01e4d7d9b8e96f54a40, disabling compactions & flushes 2023-02-11 10:00:47,893 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. after waiting 0 ms 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:47,893 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:47,893 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-1] regionserver.HRegion(1557): Region close journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Waiting for close lock at 1676109647893Disabling compacts and flushes for region at 1676109647893Disabling writes for close at 1676109647893Writing region close event to WAL at 1676109647893Closed at 1676109647893 2023-02-11 10:00:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 10:00:48,234 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,234 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1603): Closing c792e2d4f0faf7086352849794012c79, disabling compactions & flushes 2023-02-11 10:00:48,235 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,235 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,235 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. after waiting 0 ms 2023-02-11 10:00:48,235 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,235 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,235 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-0] regionserver.HRegion(1557): Region close journal for c792e2d4f0faf7086352849794012c79: Waiting for close lock at 1676109648234Disabling compacts and flushes for region at 1676109648234Disabling writes for close at 1676109648235 (+1 ms)Writing region close event to WAL at 1676109648235Closed at 1676109648235 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1603): Closing 2dba5f8399997a147e997c4d38161234, disabling compactions & flushes 2023-02-11 10:00:48,237 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. after waiting 0 ms 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,237 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,237 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676109647730-pool-2] regionserver.HRegion(1557): Region close journal for 2dba5f8399997a147e997c4d38161234: Waiting for close lock at 1676109648237Disabling compacts and flushes for region at 1676109648237Disabling writes for close at 1676109648237Writing region close event to WAL at 1676109648237Closed at 1676109648237 2023-02-11 10:00:48,240 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,242 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109648241"}]},"ts":"1676109648241"} 2023-02-11 10:00:48,248 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-11 10:00:48,249 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 10:00:48,249 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109648249"}]},"ts":"1676109648249"} 2023-02-11 10:00:48,250 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLING in hbase:meta 2023-02-11 10:00:48,261 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 10:00:48,266 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 10:00:48,266 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 10:00:48,266 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 10:00:48,266 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 10:00:48,266 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN}] 2023-02-11 10:00:48,270 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN 2023-02-11 10:00:48,270 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN 2023-02-11 10:00:48,270 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN 2023-02-11 10:00:48,271 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN 2023-02-11 10:00:48,271 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:48,271 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:48,271 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN 2023-02-11 10:00:48,272 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=false 2023-02-11 10:00:48,272 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:48,272 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN 2023-02-11 10:00:48,275 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=false 2023-02-11 10:00:48,275 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=false 2023-02-11 10:00:48,421 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:48,425 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:48,425 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:48,426 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648425"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648425"}]},"ts":"1676109648425"} 2023-02-11 10:00:48,425 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:48,425 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:48,426 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648425"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648425"}]},"ts":"1676109648425"} 2023-02-11 10:00:48,426 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648425"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648425"}]},"ts":"1676109648425"} 2023-02-11 10:00:48,425 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:48,426 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648425"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648425"}]},"ts":"1676109648425"} 2023-02-11 10:00:48,426 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648425"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648425"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648425"}]},"ts":"1676109648425"} 2023-02-11 10:00:48,428 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=208, state=RUNNABLE; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:48,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=206, state=RUNNABLE; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:48,431 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=205, state=RUNNABLE; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:48,431 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:48,432 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648431"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109648431"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109648431"}]},"ts":"1676109648431"} 2023-02-11 10:00:48,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=203, state=RUNNABLE; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:48,433 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=207, state=RUNNABLE; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:48,437 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=204, state=RUNNABLE; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 10:00:48,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:48,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1864ff4284b6d01e4d7d9b8e96f54a40, NAME => 'testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:48,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c792e2d4f0faf7086352849794012c79, NAME => 'testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:48,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,585 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,586 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2dba5f8399997a147e997c4d38161234, NAME => 'testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:48,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,587 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,589 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:48,589 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:48,589 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:48,589 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:48,589 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2dba5f8399997a147e997c4d38161234 columnFamilyName cf 2023-02-11 10:00:48,589 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c792e2d4f0faf7086352849794012c79 columnFamilyName cf 2023-02-11 10:00:48,590 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(310): Store=2dba5f8399997a147e997c4d38161234/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,590 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(310): Store=c792e2d4f0faf7086352849794012c79/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:48,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:48,597 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:48,597 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:48,597 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1864ff4284b6d01e4d7d9b8e96f54a40 columnFamilyName cf 2023-02-11 10:00:48,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,598 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(310): Store=1864ff4284b6d01e4d7d9b8e96f54a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c792e2d4f0faf7086352849794012c79; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61695790, jitterRate=-0.08066108822822571}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c792e2d4f0faf7086352849794012c79: Running coprocessor pre-open hook at 1676109648585Writing region info on filesystem at 1676109648585Initializing all the Stores at 1676109648585Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648585Cleaning up temporary data from old regions at 1676109648592 (+7 ms)Cleaning up detritus from prior splits at 1676109648593 (+1 ms)Running coprocessor post-open hooks at 1676109648598 (+5 ms)Region opened successfully at 1676109648598 2023-02-11 10:00:48,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2dba5f8399997a147e997c4d38161234; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64477824, jitterRate=-0.03920555114746094}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2dba5f8399997a147e997c4d38161234: Running coprocessor pre-open hook at 1676109648587Writing region info on filesystem at 1676109648587Initializing all the Stores at 1676109648587Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648587Cleaning up temporary data from old regions at 1676109648592 (+5 ms)Cleaning up detritus from prior splits at 1676109648593 (+1 ms)Running coprocessor post-open hooks at 1676109648599 (+6 ms)Region opened successfully at 1676109648599 2023-02-11 10:00:48,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79., pid=212, masterSystemTime=1676109648582 2023-02-11 10:00:48,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234., pid=211, masterSystemTime=1676109648583 2023-02-11 10:00:48,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:48,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:48,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 613f74e8c853d032f33c9b749dc5b78d, NAME => 'testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:48,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:48,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648602"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648602"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648602"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648602"}]},"ts":"1676109648602"} 2023-02-11 10:00:48,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:48,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:48,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5b412f238605ea2c2123a33e84ce883e, NAME => 'testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,603 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:48,603 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648603"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648603"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648603"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648603"}]},"ts":"1676109648603"} 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:48,605 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,605 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=203 2023-02-11 10:00:48,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=203, state=SUCCESS; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 172 msec 2023-02-11 10:00:48,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=205 2023-02-11 10:00:48,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=205, state=SUCCESS; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 174 msec 2023-02-11 10:00:48,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN in 340 msec 2023-02-11 10:00:48,608 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:48,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,608 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:48,608 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:48,608 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:48,608 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN in 341 msec 2023-02-11 10:00:48,608 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b412f238605ea2c2123a33e84ce883e columnFamilyName cf 2023-02-11 10:00:48,608 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 613f74e8c853d032f33c9b749dc5b78d columnFamilyName cf 2023-02-11 10:00:48,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1864ff4284b6d01e4d7d9b8e96f54a40; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72876231, jitterRate=0.08594046533107758}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Running coprocessor pre-open hook at 1676109648583Writing region info on filesystem at 1676109648583Initializing all the Stores at 1676109648584 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648584Cleaning up temporary data from old regions at 1676109648600 (+16 ms)Cleaning up detritus from prior splits at 1676109648601 (+1 ms)Running coprocessor post-open hooks at 1676109648609 (+8 ms)Region opened successfully at 1676109648609 2023-02-11 10:00:48,609 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(310): Store=5b412f238605ea2c2123a33e84ce883e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40., pid=209, masterSystemTime=1676109648580 2023-02-11 10:00:48,609 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(310): Store=613f74e8c853d032f33c9b749dc5b78d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:48,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:48,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:48,611 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f895b4386bc67b999d38d2505ce37ca, NAME => 'testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:48,611 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109648611"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648611"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648611"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648611"}]},"ts":"1676109648611"} 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,613 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:48,614 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=208 2023-02-11 10:00:48,614 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=208, state=SUCCESS; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 184 msec 2023-02-11 10:00:48,615 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:48,615 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:48,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN in 348 msec 2023-02-11 10:00:48,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:48,616 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f895b4386bc67b999d38d2505ce37ca columnFamilyName cf 2023-02-11 10:00:48,617 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(310): Store=4f895b4386bc67b999d38d2505ce37ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:48,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5b412f238605ea2c2123a33e84ce883e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60085813, jitterRate=-0.10465161502361298}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5b412f238605ea2c2123a33e84ce883e: Running coprocessor pre-open hook at 1676109648603Writing region info on filesystem at 1676109648603Initializing all the Stores at 1676109648604 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648604Cleaning up temporary data from old regions at 1676109648611 (+7 ms)Cleaning up detritus from prior splits at 1676109648612 (+1 ms)Running coprocessor post-open hooks at 1676109648617 (+5 ms)Region opened successfully at 1676109648617 2023-02-11 10:00:48,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e., pid=213, masterSystemTime=1676109648583 2023-02-11 10:00:48,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 613f74e8c853d032f33c9b749dc5b78d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65868964, jitterRate=-0.01847594976425171}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 613f74e8c853d032f33c9b749dc5b78d: Running coprocessor pre-open hook at 1676109648603Writing region info on filesystem at 1676109648603Initializing all the Stores at 1676109648604 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648604Cleaning up temporary data from old regions at 1676109648612 (+8 ms)Cleaning up detritus from prior splits at 1676109648612Running coprocessor post-open hooks at 1676109648618 (+6 ms)Region opened successfully at 1676109648618 2023-02-11 10:00:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:48,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:48,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d., pid=210, masterSystemTime=1676109648582 2023-02-11 10:00:48,620 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:48,620 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648619"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648619"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648619"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648619"}]},"ts":"1676109648619"} 2023-02-11 10:00:48,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:48,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:48,621 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:48,621 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648621"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648621"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648621"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648621"}]},"ts":"1676109648621"} 2023-02-11 10:00:48,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:48,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=207 2023-02-11 10:00:48,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=207, state=SUCCESS; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 188 msec 2023-02-11 10:00:48,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 10:00:48,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f895b4386bc67b999d38d2505ce37ca; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74613696, jitterRate=0.1118307113647461}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:48,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f895b4386bc67b999d38d2505ce37ca: Running coprocessor pre-open hook at 1676109648612Writing region info on filesystem at 1676109648612Initializing all the Stores at 1676109648612Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109648612Cleaning up temporary data from old regions at 1676109648618 (+6 ms)Cleaning up detritus from prior splits at 1676109648619 (+1 ms)Running coprocessor post-open hooks at 1676109648625 (+6 ms)Region opened successfully at 1676109648625 2023-02-11 10:00:48,625 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=206 2023-02-11 10:00:48,625 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=206, state=SUCCESS; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 193 msec 2023-02-11 10:00:48,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca., pid=214, masterSystemTime=1676109648580 2023-02-11 10:00:48,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN in 358 msec 2023-02-11 10:00:48,626 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN in 359 msec 2023-02-11 10:00:48,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:48,627 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:48,627 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:48,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109648627"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109648627"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109648627"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109648627"}]},"ts":"1676109648627"} 2023-02-11 10:00:48,630 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=204 2023-02-11 10:00:48,630 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=204, state=SUCCESS; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 192 msec 2023-02-11 10:00:48,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=204, resume processing ppid=202 2023-02-11 10:00:48,632 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN in 364 msec 2023-02-11 10:00:48,632 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 10:00:48,632 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109648632"}]},"ts":"1676109648632"} 2023-02-11 10:00:48,633 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLED in hbase:meta 2023-02-11 10:00:48,646 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 10:00:48,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676109647730 in 911 msec 2023-02-11 10:00:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-11 10:00:49,244 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 202 completed 2023-02-11 10:00:49,247 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,247 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:49,252 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:49,282 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,282 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 10:00:49,287 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109649287"}]},"ts":"1676109649287"} 2023-02-11 10:00:49,289 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLING in hbase:meta 2023-02-11 10:00:49,290 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676109647730 to state=DISABLING 2023-02-11 10:00:49,291 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN}] 2023-02-11 10:00:49,293 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN 2023-02-11 10:00:49,293 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN 2023-02-11 10:00:49,293 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN 2023-02-11 10:00:49,293 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN 2023-02-11 10:00:49,293 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN 2023-02-11 10:00:49,294 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,294 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,295 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649294"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649294"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649294"}]},"ts":"1676109649294"} 2023-02-11 10:00:49,295 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649294"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649294"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649294"}]},"ts":"1676109649294"} 2023-02-11 10:00:49,295 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,295 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,295 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649295"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649295"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649295"}]},"ts":"1676109649295"} 2023-02-11 10:00:49,295 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649295"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649295"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649295"}]},"ts":"1676109649295"} 2023-02-11 10:00:49,295 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,295 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649295"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649295"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649295"}]},"ts":"1676109649295"} 2023-02-11 10:00:49,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:49,299 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=221, state=RUNNABLE; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:49,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=218, state=RUNNABLE; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:49,304 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=217, state=RUNNABLE; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:49,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=219, state=RUNNABLE; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:49,312 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN 2023-02-11 10:00:49,313 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,313 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649313"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649313"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649313"}]},"ts":"1676109649313"} 2023-02-11 10:00:49,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:49,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:49,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5b412f238605ea2c2123a33e84ce883e, disabling compactions & flushes 2023-02-11 10:00:49,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:49,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:49,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. after waiting 0 ms 2023-02-11 10:00:49,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:49,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:49,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1864ff4284b6d01e4d7d9b8e96f54a40, disabling compactions & flushes 2023-02-11 10:00:49,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:49,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:49,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. after waiting 0 ms 2023-02-11 10:00:49,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:49,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:49,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c792e2d4f0faf7086352849794012c79, disabling compactions & flushes 2023-02-11 10:00:49,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:49,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:49,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. after waiting 0 ms 2023-02-11 10:00:49,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:49,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:49,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Waiting for close lock at 1676109649461Running coprocessor pre-close hooks at 1676109649461Disabling compacts and flushes for region at 1676109649461Disabling writes for close at 1676109649461Writing region close event to WAL at 1676109649462 (+1 ms)Running coprocessor post-close hooks at 1676109649468 (+6 ms)Closed at 1676109649468 2023-02-11 10:00:49,468 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:49,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c792e2d4f0faf7086352849794012c79: Waiting for close lock at 1676109649463Running coprocessor pre-close hooks at 1676109649463Disabling compacts and flushes for region at 1676109649463Disabling writes for close at 1676109649465 (+2 ms)Writing region close event to WAL at 1676109649465Running coprocessor post-close hooks at 1676109649469 (+4 ms)Closed at 1676109649469 2023-02-11 10:00:49,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:49,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:49,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f895b4386bc67b999d38d2505ce37ca, disabling compactions & flushes 2023-02-11 10:00:49,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:49,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:49,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. after waiting 0 ms 2023-02-11 10:00:49,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:49,471 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSED 2023-02-11 10:00:49,471 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649470"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649470"}]},"ts":"1676109649470"} 2023-02-11 10:00:49,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:49,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5b412f238605ea2c2123a33e84ce883e: Waiting for close lock at 1676109649459Running coprocessor pre-close hooks at 1676109649459Disabling compacts and flushes for region at 1676109649459Disabling writes for close at 1676109649459Writing region close event to WAL at 1676109649460 (+1 ms)Running coprocessor post-close hooks at 1676109649471 (+11 ms)Closed at 1676109649471 2023-02-11 10:00:49,472 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSED 2023-02-11 10:00:49,472 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649472"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649472"}]},"ts":"1676109649472"} 2023-02-11 10:00:49,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:49,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:49,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 613f74e8c853d032f33c9b749dc5b78d, disabling compactions & flushes 2023-02-11 10:00:49,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:49,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:49,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. after waiting 0 ms 2023-02-11 10:00:49,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:49,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:49,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:49,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2dba5f8399997a147e997c4d38161234, disabling compactions & flushes 2023-02-11 10:00:49,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:49,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:49,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. after waiting 0 ms 2023-02-11 10:00:49,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:49,480 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSED 2023-02-11 10:00:49,480 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649480"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649480"}]},"ts":"1676109649480"} 2023-02-11 10:00:49,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=221 2023-02-11 10:00:49,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=221, state=SUCCESS; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 173 msec 2023-02-11 10:00:49,484 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-11 10:00:49,484 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 160 msec 2023-02-11 10:00:49,485 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN in 192 msec 2023-02-11 10:00:49,486 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN in 193 msec 2023-02-11 10:00:49,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-11 10:00:49,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 187 msec 2023-02-11 10:00:49,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:49,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:49,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-11 10:00:49,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2dba5f8399997a147e997c4d38161234: Waiting for close lock at 1676109649475Running coprocessor pre-close hooks at 1676109649475Disabling compacts and flushes for region at 1676109649475Disabling writes for close at 1676109649475Writing region close event to WAL at 1676109649476 (+1 ms)Running coprocessor post-close hooks at 1676109649487 (+11 ms)Closed at 1676109649487 2023-02-11 10:00:49,487 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f895b4386bc67b999d38d2505ce37ca: Waiting for close lock at 1676109649470Running coprocessor pre-close hooks at 1676109649470Disabling compacts and flushes for region at 1676109649470Disabling writes for close at 1676109649470Writing region close event to WAL at 1676109649471 (+1 ms)Running coprocessor post-close hooks at 1676109649487 (+16 ms)Closed at 1676109649487 2023-02-11 10:00:49,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:49,488 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 613f74e8c853d032f33c9b749dc5b78d: Waiting for close lock at 1676109649473Running coprocessor pre-close hooks at 1676109649473Disabling compacts and flushes for region at 1676109649473Disabling writes for close at 1676109649473Writing region close event to WAL at 1676109649474 (+1 ms)Running coprocessor post-close hooks at 1676109649488 (+14 ms)Closed at 1676109649488 2023-02-11 10:00:49,489 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN in 195 msec 2023-02-11 10:00:49,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:49,489 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSED 2023-02-11 10:00:49,490 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649489"}]},"ts":"1676109649489"} 2023-02-11 10:00:49,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:49,490 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSED 2023-02-11 10:00:49,490 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649490"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649490"}]},"ts":"1676109649490"} 2023-02-11 10:00:49,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:49,491 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSED 2023-02-11 10:00:49,492 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649491"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109649491"}]},"ts":"1676109649491"} 2023-02-11 10:00:49,496 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=218 2023-02-11 10:00:49,496 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=218, state=SUCCESS; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 189 msec 2023-02-11 10:00:49,497 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=219 2023-02-11 10:00:49,497 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=219, state=SUCCESS; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 184 msec 2023-02-11 10:00:49,504 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN in 205 msec 2023-02-11 10:00:49,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=217 2023-02-11 10:00:49,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN in 206 msec 2023-02-11 10:00:49,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=217, state=SUCCESS; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 191 msec 2023-02-11 10:00:49,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=217, resume processing ppid=215 2023-02-11 10:00:49,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN in 213 msec 2023-02-11 10:00:49,511 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109649511"}]},"ts":"1676109649511"} 2023-02-11 10:00:49,513 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLED in hbase:meta 2023-02-11 10:00:49,515 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676109647730 to state=DISABLED 2023-02-11 10:00:49,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 in 234 msec 2023-02-11 10:00:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-11 10:00:49,539 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 215 completed 2023-02-11 10:00:49,539 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109649539 (current time:1676109649539). 2023-02-11 10:00:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x35c9d9a9 to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fbb5bad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:49,548 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35c9d9a9 to 127.0.0.1:64863 2023-02-11 10:00:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:49,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:49,555 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:49,556 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE; timeout=600000, timestamp=1676110249556 2023-02-11 10:00:49,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:49,556 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676109647730 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,556 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:49,557 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:49,574 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } 2023-02-11 10:00:49,581 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,581 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,584 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,585 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,585 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,585 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,585 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,590 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,590 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,591 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,592 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,593 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,596 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,600 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:49,600 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:49,609 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-11 10:00:49,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:49,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676109647730 2023-02-11 10:00:49,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109647730 2023-02-11 10:00:49,702 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109647730 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/emptySnaptb-1676109647730 2023-02-11 10:00:49,729 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676109647730 of table testCorruptedSnapshot-1676109647730 completed 2023-02-11 10:00:49,729 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109647730 2023-02-11 10:00:49,729 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/emptySnaptb-1676109647730 2023-02-11 10:00:49,729 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676109647730 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676109647730 at 1676109649556Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } at 1676109649574 (+18 ms)Consolidate snapshot: emptySnaptb-1676109647730 at 1676109649677 (+103 ms)Loading Region manifests for emptySnaptb-1676109647730 at 1676109649677Writing data manifest for emptySnaptb-1676109647730 at 1676109649684 (+7 ms)Verifying snapshot: emptySnaptb-1676109647730 at 1676109649698 (+14 ms)Snapshot emptySnaptb-1676109647730 of table testCorruptedSnapshot-1676109647730 completed at 1676109649729 (+31 ms) 2023-02-11 10:00:49,731 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:49,732 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE in 178 msec 2023-02-11 10:00:49,807 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } is done 2023-02-11 10:00:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:49,808 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,809 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 10:00:49,812 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:49,813 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109649812"}]},"ts":"1676109649812"} 2023-02-11 10:00:49,814 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLING in hbase:meta 2023-02-11 10:00:49,816 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:49,816 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN}] 2023-02-11 10:00:49,820 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN 2023-02-11 10:00:49,820 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN 2023-02-11 10:00:49,820 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN 2023-02-11 10:00:49,821 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN 2023-02-11 10:00:49,821 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN 2023-02-11 10:00:49,821 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,821 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,821 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:49,821 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:49,821 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,821 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:49,822 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,822 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:49,822 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,822 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:49,822 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN 2023-02-11 10:00:49,823 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,823 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:49,971 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:49,972 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,972 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,972 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:49,972 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:49,972 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649972"}]},"ts":"1676109649972"} 2023-02-11 10:00:49,972 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,972 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649972"}]},"ts":"1676109649972"} 2023-02-11 10:00:49,972 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649972"}]},"ts":"1676109649972"} 2023-02-11 10:00:49,972 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109649972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649972"}]},"ts":"1676109649972"} 2023-02-11 10:00:49,972 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649972"}]},"ts":"1676109649972"} 2023-02-11 10:00:49,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=233, state=RUNNABLE; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:49,974 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=235, state=RUNNABLE; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:49,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=231, state=RUNNABLE; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:49,976 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=230, state=RUNNABLE; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:49,976 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=232, state=RUNNABLE; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:49,978 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:49,978 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109649978"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109649978"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109649978"}]},"ts":"1676109649978"} 2023-02-11 10:00:49,979 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=234, state=RUNNABLE; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:50,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 10:00:50,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5b412f238605ea2c2123a33e84ce883e, NAME => 'testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:50,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c792e2d4f0faf7086352849794012c79, NAME => 'testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1864ff4284b6d01e4d7d9b8e96f54a40, NAME => 'testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,133 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,133 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,133 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,133 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:50,134 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:50,134 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:50,134 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:50,134 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:50,134 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:50,134 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1864ff4284b6d01e4d7d9b8e96f54a40 columnFamilyName cf 2023-02-11 10:00:50,134 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b412f238605ea2c2123a33e84ce883e columnFamilyName cf 2023-02-11 10:00:50,134 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c792e2d4f0faf7086352849794012c79 columnFamilyName cf 2023-02-11 10:00:50,135 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(310): Store=c792e2d4f0faf7086352849794012c79/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,135 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(310): Store=5b412f238605ea2c2123a33e84ce883e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5b412f238605ea2c2123a33e84ce883e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66907576, jitterRate=-0.002999424934387207}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5b412f238605ea2c2123a33e84ce883e: Running coprocessor pre-open hook at 1676109650131Writing region info on filesystem at 1676109650131Initializing all the Stores at 1676109650132 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650132Cleaning up temporary data from old regions at 1676109650137 (+5 ms)Cleaning up detritus from prior splits at 1676109650137Running coprocessor post-open hooks at 1676109650140 (+3 ms)Region opened successfully at 1676109650140 2023-02-11 10:00:50,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c792e2d4f0faf7086352849794012c79; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66181894, jitterRate=-0.013812929391860962}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c792e2d4f0faf7086352849794012c79: Running coprocessor pre-open hook at 1676109650131Writing region info on filesystem at 1676109650131Initializing all the Stores at 1676109650132 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650132Cleaning up temporary data from old regions at 1676109650137 (+5 ms)Cleaning up detritus from prior splits at 1676109650138 (+1 ms)Running coprocessor post-open hooks at 1676109650140 (+2 ms)Region opened successfully at 1676109650140 2023-02-11 10:00:50,140 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(310): Store=1864ff4284b6d01e4d7d9b8e96f54a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e., pid=241, masterSystemTime=1676109650127 2023-02-11 10:00:50,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79., pid=239, masterSystemTime=1676109650127 2023-02-11 10:00:50,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2dba5f8399997a147e997c4d38161234, NAME => 'testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:50,145 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1864ff4284b6d01e4d7d9b8e96f54a40; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67262953, jitterRate=0.0022961050271987915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,145 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650144"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650144"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650144"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650144"}]},"ts":"1676109650144"} 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Running coprocessor pre-open hook at 1676109650132Writing region info on filesystem at 1676109650132Initializing all the Stores at 1676109650132Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650133 (+1 ms)Cleaning up temporary data from old regions at 1676109650142 (+9 ms)Cleaning up detritus from prior splits at 1676109650142Running coprocessor post-open hooks at 1676109650145 (+3 ms)Region opened successfully at 1676109650145 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,145 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:50,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,145 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650145"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650145"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650145"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650145"}]},"ts":"1676109650145"} 2023-02-11 10:00:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 613f74e8c853d032f33c9b749dc5b78d, NAME => 'testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:50,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40., pid=237, masterSystemTime=1676109650127 2023-02-11 10:00:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,147 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f895b4386bc67b999d38d2505ce37ca, NAME => 'testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:50,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:50,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,148 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,148 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:50,148 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:50,148 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:50,148 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650148"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650148"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650148"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650148"}]},"ts":"1676109650148"} 2023-02-11 10:00:50,148 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2dba5f8399997a147e997c4d38161234 columnFamilyName cf 2023-02-11 10:00:50,149 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:50,149 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:50,149 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,149 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=234 2023-02-11 10:00:50,149 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=234, state=SUCCESS; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 168 msec 2023-02-11 10:00:50,149 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 613f74e8c853d032f33c9b749dc5b78d columnFamilyName cf 2023-02-11 10:00:50,149 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(310): Store=2dba5f8399997a147e997c4d38161234/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,150 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=230 2023-02-11 10:00:50,150 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(310): Store=613f74e8c853d032f33c9b749dc5b78d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,150 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:50,150 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:50,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,150 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=230, state=SUCCESS; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 171 msec 2023-02-11 10:00:50,151 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f895b4386bc67b999d38d2505ce37ca columnFamilyName cf 2023-02-11 10:00:50,151 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN in 333 msec 2023-02-11 10:00:50,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,152 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN in 334 msec 2023-02-11 10:00:50,152 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(310): Store=4f895b4386bc67b999d38d2505ce37ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:50,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=235 2023-02-11 10:00:50,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=235, state=SUCCESS; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 176 msec 2023-02-11 10:00:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,153 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN in 336 msec 2023-02-11 10:00:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2dba5f8399997a147e997c4d38161234; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64840613, jitterRate=-0.03379957377910614}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2dba5f8399997a147e997c4d38161234: Running coprocessor pre-open hook at 1676109650145Writing region info on filesystem at 1676109650145Initializing all the Stores at 1676109650146 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650146Cleaning up temporary data from old regions at 1676109650152 (+6 ms)Cleaning up detritus from prior splits at 1676109650152Running coprocessor post-open hooks at 1676109650155 (+3 ms)Region opened successfully at 1676109650155 2023-02-11 10:00:50,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234., pid=240, masterSystemTime=1676109650127 2023-02-11 10:00:50,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 613f74e8c853d032f33c9b749dc5b78d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60421196, jitterRate=-0.09965401887893677}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f895b4386bc67b999d38d2505ce37ca; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62398113, jitterRate=-0.07019565999507904}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:50,157 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:50,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 613f74e8c853d032f33c9b749dc5b78d: Running coprocessor pre-open hook at 1676109650146Writing region info on filesystem at 1676109650146Initializing all the Stores at 1676109650147 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650147Cleaning up temporary data from old regions at 1676109650153 (+6 ms)Cleaning up detritus from prior splits at 1676109650153Running coprocessor post-open hooks at 1676109650157 (+4 ms)Region opened successfully at 1676109650157 2023-02-11 10:00:50,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f895b4386bc67b999d38d2505ce37ca: Running coprocessor pre-open hook at 1676109650148Writing region info on filesystem at 1676109650148Initializing all the Stores at 1676109650149 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109650149Cleaning up temporary data from old regions at 1676109650154 (+5 ms)Cleaning up detritus from prior splits at 1676109650154Running coprocessor post-open hooks at 1676109650157 (+3 ms)Region opened successfully at 1676109650157 2023-02-11 10:00:50,157 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650157"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650157"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650157"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650157"}]},"ts":"1676109650157"} 2023-02-11 10:00:50,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca., pid=238, masterSystemTime=1676109650127 2023-02-11 10:00:50,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d., pid=236, masterSystemTime=1676109650127 2023-02-11 10:00:50,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,159 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:50,159 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650159"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650159"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650159"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650159"}]},"ts":"1676109650159"} 2023-02-11 10:00:50,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,160 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:50,160 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650160"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109650160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109650160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109650160"}]},"ts":"1676109650160"} 2023-02-11 10:00:50,161 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=232 2023-02-11 10:00:50,161 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=232, state=SUCCESS; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 183 msec 2023-02-11 10:00:50,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN in 345 msec 2023-02-11 10:00:50,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=231 2023-02-11 10:00:50,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=231, state=SUCCESS; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 186 msec 2023-02-11 10:00:50,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=233 2023-02-11 10:00:50,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=233, state=SUCCESS; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 189 msec 2023-02-11 10:00:50,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN in 347 msec 2023-02-11 10:00:50,165 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=233, resume processing ppid=229 2023-02-11 10:00:50,165 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN in 347 msec 2023-02-11 10:00:50,165 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109650165"}]},"ts":"1676109650165"} 2023-02-11 10:00:50,168 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLED in hbase:meta 2023-02-11 10:00:50,170 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676109647730' was successfully enabled. 2023-02-11 10:00:50,171 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676109647730 in 361 msec 2023-02-11 10:00:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-11 10:00:50,565 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 229 completed 2023-02-11 10:00:50,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:50,583 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:50,583 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,583 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:50,598 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:50,599 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:50,600 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:50,601 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:50,602 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:50,607 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676109647730 2023-02-11 10:00:50,607 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676109647730 2023-02-11 10:00:50,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 10:00:50,611 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109650611"}]},"ts":"1676109650611"} 2023-02-11 10:00:50,612 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLING in hbase:meta 2023-02-11 10:00:50,614 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676109647730 to state=DISABLING 2023-02-11 10:00:50,615 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN}] 2023-02-11 10:00:50,615 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN 2023-02-11 10:00:50,615 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN 2023-02-11 10:00:50,616 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN 2023-02-11 10:00:50,616 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN 2023-02-11 10:00:50,616 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN 2023-02-11 10:00:50,616 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:50,617 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:50,617 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650616"}]},"ts":"1676109650616"} 2023-02-11 10:00:50,616 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:50,616 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:50,617 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650616"}]},"ts":"1676109650616"} 2023-02-11 10:00:50,616 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:50,617 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650616"}]},"ts":"1676109650616"} 2023-02-11 10:00:50,617 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650616"}]},"ts":"1676109650616"} 2023-02-11 10:00:50,617 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650616"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650616"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650616"}]},"ts":"1676109650616"} 2023-02-11 10:00:50,618 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=244, state=RUNNABLE; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:50,618 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=246, state=RUNNABLE; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:50,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=248, state=RUNNABLE; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:50,620 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=243, state=RUNNABLE; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:50,620 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=245, state=RUNNABLE; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:50,621 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN 2023-02-11 10:00:50,621 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:50,621 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650621"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109650621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109650621"}]},"ts":"1676109650621"} 2023-02-11 10:00:50,622 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:50,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f895b4386bc67b999d38d2505ce37ca, disabling compactions & flushes 2023-02-11 10:00:50,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. after waiting 0 ms 2023-02-11 10:00:50,770 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4f895b4386bc67b999d38d2505ce37ca 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-11 10:00:50,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 613f74e8c853d032f33c9b749dc5b78d, disabling compactions & flushes 2023-02-11 10:00:50,772 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. after waiting 1 ms 2023-02-11 10:00:50,773 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,773 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 613f74e8c853d032f33c9b749dc5b78d 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-11 10:00:50,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2dba5f8399997a147e997c4d38161234, disabling compactions & flushes 2023-02-11 10:00:50,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. after waiting 0 ms 2023-02-11 10:00:50,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2dba5f8399997a147e997c4d38161234 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 10:00:50,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111 2023-02-11 10:00:50,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa 2023-02-11 10:00:50,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28 2023-02-11 10:00:50,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/.tmp/cf/6cbc83dae62c40d18fd9754e3619dc2e 2023-02-11 10:00:50,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/.tmp/cf/f6d039d021da4ea7a0e9bd3d87936f52 2023-02-11 10:00:50,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 10:00:50,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/.tmp/cf/6cbc83dae62c40d18fd9754e3619dc2e as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e 2023-02-11 10:00:50,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/.tmp/cf/f6d039d021da4ea7a0e9bd3d87936f52 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52 2023-02-11 10:00:50,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/.tmp/cf/6b78918e0a344151b41e0b5d928f6afa 2023-02-11 10:00:50,867 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e, entries=23, sequenceid=8, filesize=8.8 K 2023-02-11 10:00:50,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 613f74e8c853d032f33c9b749dc5b78d in 94ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676109647730' 2023-02-11 10:00:50,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52, entries=40, sequenceid=8, filesize=11.8 K 2023-02-11 10:00:50,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/.tmp/cf/6b78918e0a344151b41e0b5d928f6afa as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa 2023-02-11 10:00:50,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 4f895b4386bc67b999d38d2505ce37ca in 99ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:50,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 613f74e8c853d032f33c9b749dc5b78d: Waiting for close lock at 1676109650771Running coprocessor pre-close hooks at 1676109650771Disabling compacts and flushes for region at 1676109650771Disabling writes for close at 1676109650773 (+2 ms)Obtaining lock to block concurrent updates at 1676109650773Preparing flush snapshotting stores in 613f74e8c853d032f33c9b749dc5b78d at 1676109650773Finished memstore snapshotting testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676109650773Flushing stores of testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. at 1676109650773Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: creating writer at 1676109650775 (+2 ms)Flushing mob file 613f74e8c853d032f33c9b749dc5b78d/cf: appending metadata at 1676109650800 (+25 ms)Flushing mob file 613f74e8c853d032f33c9b749dc5b78d/cf: closing flushed file at 1676109650800Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: appending metadata at 1676109650844 (+44 ms)Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: closing flushed file at 1676109650844Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: reopening flushed file at 1676109650864 (+20 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for 613f74e8c853d032f33c9b749dc5b78d in 94ms, sequenceid=8, compaction requested=false at 1676109650868 (+4 ms)Writing region close event to WAL at 1676109650869 (+1 ms)Running coprocessor post-close hooks at 1676109650872 (+3 ms)Closed at 1676109650872 2023-02-11 10:00:50,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa, entries=26, sequenceid=8, filesize=9.4 K 2023-02-11 10:00:50,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:50,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f895b4386bc67b999d38d2505ce37ca: Waiting for close lock at 1676109650770Running coprocessor pre-close hooks at 1676109650770Disabling compacts and flushes for region at 1676109650770Disabling writes for close at 1676109650770Obtaining lock to block concurrent updates at 1676109650770Preparing flush snapshotting stores in 4f895b4386bc67b999d38d2505ce37ca at 1676109650770Finished memstore snapshotting testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca., syncing WAL and waiting on mvcc, flushsize=dataSize=2678, getHeapSize=6000, getOffHeapSize=0, getCellsCount=40 at 1676109650771 (+1 ms)Flushing stores of testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. at 1676109650771Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: creating writer at 1676109650772 (+1 ms)Flushing mob file 4f895b4386bc67b999d38d2505ce37ca/cf: appending metadata at 1676109650803 (+31 ms)Flushing mob file 4f895b4386bc67b999d38d2505ce37ca/cf: closing flushed file at 1676109650803Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: appending metadata at 1676109650843 (+40 ms)Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: closing flushed file at 1676109650843Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: reopening flushed file at 1676109650866 (+23 ms)Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for 4f895b4386bc67b999d38d2505ce37ca in 99ms, sequenceid=8, compaction requested=false at 1676109650869 (+3 ms)Writing region close event to WAL at 1676109650870 (+1 ms)Running coprocessor post-close hooks at 1676109650873 (+3 ms)Closed at 1676109650873 2023-02-11 10:00:50,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 2dba5f8399997a147e997c4d38161234 in 93ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:50,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c792e2d4f0faf7086352849794012c79, disabling compactions & flushes 2023-02-11 10:00:50,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. after waiting 0 ms 2023-02-11 10:00:50,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c792e2d4f0faf7086352849794012c79 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 10:00:50,874 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSED 2023-02-11 10:00:50,874 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650874"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650874"}]},"ts":"1676109650874"} 2023-02-11 10:00:50,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:50,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1864ff4284b6d01e4d7d9b8e96f54a40, disabling compactions & flushes 2023-02-11 10:00:50,875 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSED 2023-02-11 10:00:50,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,875 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650875"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650875"}]},"ts":"1676109650875"} 2023-02-11 10:00:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. after waiting 0 ms 2023-02-11 10:00:50,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1864ff4284b6d01e4d7d9b8e96f54a40 1/1 column families, dataSize=23.03 KB heapSize=49.75 KB 2023-02-11 10:00:50,880 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:50,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2dba5f8399997a147e997c4d38161234: Waiting for close lock at 1676109650781Running coprocessor pre-close hooks at 1676109650781Disabling compacts and flushes for region at 1676109650781Disabling writes for close at 1676109650781Obtaining lock to block concurrent updates at 1676109650781Preparing flush snapshotting stores in 2dba5f8399997a147e997c4d38161234 at 1676109650781Finished memstore snapshotting testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676109650781Flushing stores of testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. at 1676109650782 (+1 ms)Flushing 2dba5f8399997a147e997c4d38161234/cf: creating writer at 1676109650793 (+11 ms)Flushing mob file 2dba5f8399997a147e997c4d38161234/cf: appending metadata at 1676109650813 (+20 ms)Flushing mob file 2dba5f8399997a147e997c4d38161234/cf: closing flushed file at 1676109650813Flushing 2dba5f8399997a147e997c4d38161234/cf: appending metadata at 1676109650849 (+36 ms)Flushing 2dba5f8399997a147e997c4d38161234/cf: closing flushed file at 1676109650849Flushing 2dba5f8399997a147e997c4d38161234/cf: reopening flushed file at 1676109650870 (+21 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 2dba5f8399997a147e997c4d38161234 in 93ms, sequenceid=8, compaction requested=false at 1676109650874 (+4 ms)Writing region close event to WAL at 1676109650874Running coprocessor post-close hooks at 1676109650880 (+6 ms)Closed at 1676109650880 2023-02-11 10:00:50,881 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=246 2023-02-11 10:00:50,881 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=246, state=SUCCESS; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 261 msec 2023-02-11 10:00:50,882 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=244 2023-02-11 10:00:50,882 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=244, state=SUCCESS; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 261 msec 2023-02-11 10:00:50,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:50,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN in 266 msec 2023-02-11 10:00:50,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5b412f238605ea2c2123a33e84ce883e, disabling compactions & flushes 2023-02-11 10:00:50,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. after waiting 0 ms 2023-02-11 10:00:50,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5b412f238605ea2c2123a33e84ce883e 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-11 10:00:50,883 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSED 2023-02-11 10:00:50,884 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650883"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650883"}]},"ts":"1676109650883"} 2023-02-11 10:00:50,884 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN in 267 msec 2023-02-11 10:00:50,886 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=245 2023-02-11 10:00:50,886 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=245, state=SUCCESS; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 265 msec 2023-02-11 10:00:50,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN in 271 msec 2023-02-11 10:00:50,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703 2023-02-11 10:00:50,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac 2023-02-11 10:00:50,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8 2023-02-11 10:00:50,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/.tmp/cf/878eb54fdc21427ca36fd5e2f4a7890f 2023-02-11 10:00:50,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/.tmp/cf/878eb54fdc21427ca36fd5e2f4a7890f as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f 2023-02-11 10:00:50,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f, entries=29, sequenceid=8, filesize=9.9 K 2023-02-11 10:00:50,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for c792e2d4f0faf7086352849794012c79 in 72ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/.tmp/cf/2c0bfd25d86241e7822bae53c725c765 2023-02-11 10:00:50,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:50,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c792e2d4f0faf7086352849794012c79: Waiting for close lock at 1676109650874Running coprocessor pre-close hooks at 1676109650874Disabling compacts and flushes for region at 1676109650874Disabling writes for close at 1676109650874Obtaining lock to block concurrent updates at 1676109650874Preparing flush snapshotting stores in c792e2d4f0faf7086352849794012c79 at 1676109650874Finished memstore snapshotting testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676109650874Flushing stores of testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. at 1676109650875 (+1 ms)Flushing c792e2d4f0faf7086352849794012c79/cf: creating writer at 1676109650876 (+1 ms)Flushing mob file c792e2d4f0faf7086352849794012c79/cf: appending metadata at 1676109650882 (+6 ms)Flushing mob file c792e2d4f0faf7086352849794012c79/cf: closing flushed file at 1676109650882Flushing c792e2d4f0faf7086352849794012c79/cf: appending metadata at 1676109650896 (+14 ms)Flushing c792e2d4f0faf7086352849794012c79/cf: closing flushed file at 1676109650896Flushing c792e2d4f0faf7086352849794012c79/cf: reopening flushed file at 1676109650942 (+46 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for c792e2d4f0faf7086352849794012c79 in 72ms, sequenceid=8, compaction requested=false at 1676109650946 (+4 ms)Writing region close event to WAL at 1676109650950 (+4 ms)Running coprocessor post-close hooks at 1676109650957 (+7 ms)Closed at 1676109650957 2023-02-11 10:00:50,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/.tmp/cf/89bec9e0b6e2406e9cb1c17ab313fc06 2023-02-11 10:00:50,960 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:50,960 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSED 2023-02-11 10:00:50,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/.tmp/cf/2c0bfd25d86241e7822bae53c725c765 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765 2023-02-11 10:00:50,960 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650960"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650960"}]},"ts":"1676109650960"} 2023-02-11 10:00:50,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/.tmp/cf/89bec9e0b6e2406e9cb1c17ab313fc06 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06 2023-02-11 10:00:50,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=243 2023-02-11 10:00:50,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=243, state=SUCCESS; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 342 msec 2023-02-11 10:00:50,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06, entries=30, sequenceid=8, filesize=10.1 K 2023-02-11 10:00:50,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765, entries=352, sequenceid=8, filesize=65.6 K 2023-02-11 10:00:50,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN in 353 msec 2023-02-11 10:00:50,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.03 KB/23580, heapSize ~49.73 KB/50928, currentSize=0 B/0 for 1864ff4284b6d01e4d7d9b8e96f54a40 in 94ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5b412f238605ea2c2123a33e84ce883e in 86ms, sequenceid=8, compaction requested=false 2023-02-11 10:00:50,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-11 10:00:50,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:50,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:50,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5b412f238605ea2c2123a33e84ce883e: Waiting for close lock at 1676109650883Running coprocessor pre-close hooks at 1676109650883Disabling compacts and flushes for region at 1676109650883Disabling writes for close at 1676109650883Obtaining lock to block concurrent updates at 1676109650883Preparing flush snapshotting stores in 5b412f238605ea2c2123a33e84ce883e at 1676109650883Finished memstore snapshotting testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676109650883Flushing stores of testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. at 1676109650884 (+1 ms)Flushing 5b412f238605ea2c2123a33e84ce883e/cf: creating writer at 1676109650884Flushing mob file 5b412f238605ea2c2123a33e84ce883e/cf: appending metadata at 1676109650894 (+10 ms)Flushing mob file 5b412f238605ea2c2123a33e84ce883e/cf: closing flushed file at 1676109650894Flushing 5b412f238605ea2c2123a33e84ce883e/cf: appending metadata at 1676109650933 (+39 ms)Flushing 5b412f238605ea2c2123a33e84ce883e/cf: closing flushed file at 1676109650933Flushing 5b412f238605ea2c2123a33e84ce883e/cf: reopening flushed file at 1676109650966 (+33 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5b412f238605ea2c2123a33e84ce883e in 86ms, sequenceid=8, compaction requested=false at 1676109650970 (+4 ms)Writing region close event to WAL at 1676109650970Running coprocessor post-close hooks at 1676109650973 (+3 ms)Closed at 1676109650973 2023-02-11 10:00:50,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Waiting for close lock at 1676109650875Running coprocessor pre-close hooks at 1676109650875Disabling compacts and flushes for region at 1676109650875Disabling writes for close at 1676109650875Obtaining lock to block concurrent updates at 1676109650875Preparing flush snapshotting stores in 1864ff4284b6d01e4d7d9b8e96f54a40 at 1676109650875Finished memstore snapshotting testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40., syncing WAL and waiting on mvcc, flushsize=dataSize=23580, getHeapSize=50928, getOffHeapSize=0, getCellsCount=352 at 1676109650875Flushing stores of testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. at 1676109650876 (+1 ms)Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: creating writer at 1676109650878 (+2 ms)Flushing mob file 1864ff4284b6d01e4d7d9b8e96f54a40/cf: appending metadata at 1676109650885 (+7 ms)Flushing mob file 1864ff4284b6d01e4d7d9b8e96f54a40/cf: closing flushed file at 1676109650885Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: appending metadata at 1676109650915 (+30 ms)Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: closing flushed file at 1676109650915Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: reopening flushed file at 1676109650961 (+46 ms)Finished flush of dataSize ~23.03 KB/23580, heapSize ~49.73 KB/50928, currentSize=0 B/0 for 1864ff4284b6d01e4d7d9b8e96f54a40 in 94ms, sequenceid=8, compaction requested=false at 1676109650970 (+9 ms)Writing region close event to WAL at 1676109650970Running coprocessor post-close hooks at 1676109650973 (+3 ms)Closed at 1676109650973 2023-02-11 10:00:50,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:50,975 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSED 2023-02-11 10:00:50,975 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109650975"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650975"}]},"ts":"1676109650975"} 2023-02-11 10:00:50,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:50,975 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSED 2023-02-11 10:00:50,975 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109650975"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109650975"}]},"ts":"1676109650975"} 2023-02-11 10:00:50,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-11 10:00:50,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 354 msec 2023-02-11 10:00:50,978 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=248 2023-02-11 10:00:50,978 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=248, state=SUCCESS; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 357 msec 2023-02-11 10:00:50,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN in 362 msec 2023-02-11 10:00:50,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=248, resume processing ppid=242 2023-02-11 10:00:50,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN in 363 msec 2023-02-11 10:00:50,980 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109650980"}]},"ts":"1676109650980"} 2023-02-11 10:00:50,981 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLED in hbase:meta 2023-02-11 10:00:50,982 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676109647730 to state=DISABLED 2023-02-11 10:00:50,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 in 376 msec 2023-02-11 10:00:51,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-11 10:00:51,364 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 242 completed 2023-02-11 10:00:51,365 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676109651365 (current time:1676109651365). 2023-02-11 10:00:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-11 10:00:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(139): Connect 0x0bc602af to 127.0.0.1:64863 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 10:00:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@703996ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 10:00:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 10:00:51,376 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54950, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 10:00:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bc602af to 127.0.0.1:64863 2023-02-11 10:00:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-11 10:00:51,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:51,380 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } 2023-02-11 10:00:51,381 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE; timeout=600000, timestamp=1676110251381 2023-02-11 10:00:51,381 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676109647730 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:51,382 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-11 10:00:51,382 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-11 10:00:51,396 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } 2023-02-11 10:00:51,400 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,400 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,400 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,400 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,402 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,403 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,404 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-11 10:00:51,404 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-11 10:00:51,405 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f] hfiles 2023-02-11 10:00:51,405 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e] hfiles 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52] hfiles 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765] hfiles 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06] hfiles 2023-02-11 10:00:51,406 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06 2023-02-11 10:00:51,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa] hfiles 2023-02-11 10:00:51,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa 2023-02-11 10:00:51,411 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa] hfiles 2023-02-11 10:00:51,411 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8 2023-02-11 10:00:51,411 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111 2023-02-11 10:00:51,412 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28 2023-02-11 10:00:51,412 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703 2023-02-11 10:00:51,412 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac 2023-02-11 10:00:51,412 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa 2023-02-11 10:00:51,424 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } as finished. 2023-02-11 10:00:51,425 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676109647730 2023-02-11 10:00:51,425 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109647730 2023-02-11 10:00:51,443 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109647730 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/snaptb0-1676109647730 2023-02-11 10:00:51,460 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676109647730 of table testCorruptedSnapshot-1676109647730 completed 2023-02-11 10:00:51,460 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109647730 2023-02-11 10:00:51,460 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.hbase-snapshot/.tmp/snaptb0-1676109647730 2023-02-11 10:00:51,460 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676109647730 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676109647730 at 1676109651381Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=DISABLED ttl=0 } at 1676109651396 (+15 ms)Consolidate snapshot: snaptb0-1676109647730 at 1676109651425 (+29 ms)Loading Region manifests for snaptb0-1676109647730 at 1676109651425Writing data manifest for snaptb0-1676109647730 at 1676109651428 (+3 ms)Verifying snapshot: snaptb0-1676109647730 at 1676109651435 (+7 ms)Snapshot snaptb0-1676109647730 of table testCorruptedSnapshot-1676109647730 completed at 1676109651460 (+25 ms) 2023-02-11 10:00:51,461 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE 2023-02-11 10:00:51,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676109647730, type=EXCLUSIVE in 83 msec 2023-02-11 10:00:51,632 DEBUG [Listener at localhost/42305] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-11 10:00:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 } is done 2023-02-11 10:00:51,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676109647730 table=testCorruptedSnapshot-1676109647730 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-11 10:00:51,633 INFO [Listener at localhost/42305] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676109647730 2023-02-11 10:00:51,634 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676109647730 2023-02-11 10:00:51,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:51,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 10:00:51,637 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:51,637 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109651637"}]},"ts":"1676109651637"} 2023-02-11 10:00:51,638 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLING in hbase:meta 2023-02-11 10:00:51,640 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-11 10:00:51,640 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN}] 2023-02-11 10:00:51,641 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN 2023-02-11 10:00:51,641 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN 2023-02-11 10:00:51,641 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN 2023-02-11 10:00:51,641 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN 2023-02-11 10:00:51,642 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN 2023-02-11 10:00:51,642 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,642 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:51,642 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,642 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:51,643 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,643 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,643 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,643 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:51,643 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35443,1676109614327; forceNewPlan=false, retain=true 2023-02-11 10:00:51,643 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41193,1676109614244; forceNewPlan=false, retain=true 2023-02-11 10:00:51,643 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN 2023-02-11 10:00:51,644 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,644 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41567,1676109614293; forceNewPlan=false, retain=true 2023-02-11 10:00:51,792 INFO [jenkins-hbase4:42723] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-11 10:00:51,792 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,792 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,792 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651792"}]},"ts":"1676109651792"} 2023-02-11 10:00:51,792 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,792 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,792 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,793 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651792"}]},"ts":"1676109651792"} 2023-02-11 10:00:51,793 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109651792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651792"}]},"ts":"1676109651792"} 2023-02-11 10:00:51,793 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651792"}]},"ts":"1676109651792"} 2023-02-11 10:00:51,793 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109651792"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651792"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651792"}]},"ts":"1676109651792"} 2023-02-11 10:00:51,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=258, state=RUNNABLE; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:51,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=260, state=RUNNABLE; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:51,795 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=257, state=RUNNABLE; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:51,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:51,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=262, state=RUNNABLE; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:51,799 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,799 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651799"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109651799"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109651799"}]},"ts":"1676109651799"} 2023-02-11 10:00:51,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=261, state=RUNNABLE; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 10:00:51,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 613f74e8c853d032f33c9b749dc5b78d, NAME => 'testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4f895b4386bc67b999d38d2505ce37ca, NAME => 'testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.', STARTKEY => '1', ENDKEY => '2'} 2023-02-11 10:00:51,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5b412f238605ea2c2123a33e84ce883e, NAME => 'testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.', STARTKEY => '4', ENDKEY => '5'} 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,951 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,952 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,952 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:51,952 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf 2023-02-11 10:00:51,952 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,952 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:51,953 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf 2023-02-11 10:00:51,953 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 613f74e8c853d032f33c9b749dc5b78d columnFamilyName cf 2023-02-11 10:00:51,953 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f895b4386bc67b999d38d2505ce37ca columnFamilyName cf 2023-02-11 10:00:51,953 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:51,953 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf 2023-02-11 10:00:51,953 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5b412f238605ea2c2123a33e84ce883e columnFamilyName cf 2023-02-11 10:00:51,956 DEBUG [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e 2023-02-11 10:00:51,957 INFO [StoreOpener-613f74e8c853d032f33c9b749dc5b78d-1] regionserver.HStore(310): Store=613f74e8c853d032f33c9b749dc5b78d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,957 DEBUG [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52 2023-02-11 10:00:51,957 INFO [StoreOpener-4f895b4386bc67b999d38d2505ce37ca-1] regionserver.HStore(310): Store=4f895b4386bc67b999d38d2505ce37ca/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,957 DEBUG [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06 2023-02-11 10:00:51,957 INFO [StoreOpener-5b412f238605ea2c2123a33e84ce883e-1] regionserver.HStore(310): Store=5b412f238605ea2c2123a33e84ce883e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:51,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:51,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:51,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 613f74e8c853d032f33c9b749dc5b78d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69183207, jitterRate=0.03091011941432953}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 613f74e8c853d032f33c9b749dc5b78d: Running coprocessor pre-open hook at 1676109651950Writing region info on filesystem at 1676109651950Initializing all the Stores at 1676109651951 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651951Cleaning up temporary data from old regions at 1676109651960 (+9 ms)Cleaning up detritus from prior splits at 1676109651960Running coprocessor post-open hooks at 1676109651963 (+3 ms)Region opened successfully at 1676109651963 2023-02-11 10:00:51,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 4f895b4386bc67b999d38d2505ce37ca; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75147495, jitterRate=0.11978493630886078}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 5b412f238605ea2c2123a33e84ce883e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64309644, jitterRate=-0.041711628437042236}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 4f895b4386bc67b999d38d2505ce37ca: Running coprocessor pre-open hook at 1676109651951Writing region info on filesystem at 1676109651951Initializing all the Stores at 1676109651951Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651951Cleaning up temporary data from old regions at 1676109651960 (+9 ms)Cleaning up detritus from prior splits at 1676109651960Running coprocessor post-open hooks at 1676109651963 (+3 ms)Region opened successfully at 1676109651963 2023-02-11 10:00:51,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 5b412f238605ea2c2123a33e84ce883e: Running coprocessor pre-open hook at 1676109651951Writing region info on filesystem at 1676109651951Initializing all the Stores at 1676109651952 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651952Cleaning up temporary data from old regions at 1676109651960 (+8 ms)Cleaning up detritus from prior splits at 1676109651960Running coprocessor post-open hooks at 1676109651963 (+3 ms)Region opened successfully at 1676109651963 2023-02-11 10:00:51,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e., pid=268, masterSystemTime=1676109651948 2023-02-11 10:00:51,964 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d., pid=264, masterSystemTime=1676109651947 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:51,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:51,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2dba5f8399997a147e997c4d38161234, NAME => 'testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.', STARTKEY => '2', ENDKEY => '3'} 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,966 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,966 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651966"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651966"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651966"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651966"}]},"ts":"1676109651966"} 2023-02-11 10:00:51,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:51,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:51,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:51,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c792e2d4f0faf7086352849794012c79, NAME => 'testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.', STARTKEY => '', ENDKEY => '1'} 2023-02-11 10:00:51,966 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,967 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,967 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651966"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651966"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651966"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651966"}]},"ts":"1676109651966"} 2023-02-11 10:00:51,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,968 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:51,968 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf 2023-02-11 10:00:51,968 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca., pid=263, masterSystemTime=1676109651947 2023-02-11 10:00:51,969 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2dba5f8399997a147e997c4d38161234 columnFamilyName cf 2023-02-11 10:00:51,969 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:51,969 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf 2023-02-11 10:00:51,969 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c792e2d4f0faf7086352849794012c79 columnFamilyName cf 2023-02-11 10:00:51,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:51,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:51,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1864ff4284b6d01e4d7d9b8e96f54a40, NAME => 'testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.', STARTKEY => '5', ENDKEY => ''} 2023-02-11 10:00:51,973 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=261 2023-02-11 10:00:51,973 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=261, state=SUCCESS; OpenRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 168 msec 2023-02-11 10:00:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676109647730 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 10:00:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,973 DEBUG [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f 2023-02-11 10:00:51,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=260 2023-02-11 10:00:51,974 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,974 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651973"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651973"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651973"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651973"}]},"ts":"1676109651973"} 2023-02-11 10:00:51,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=260, state=SUCCESS; OpenRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 173 msec 2023-02-11 10:00:51,974 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, ASSIGN in 333 msec 2023-02-11 10:00:51,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, ASSIGN in 334 msec 2023-02-11 10:00:51,976 INFO [StoreOpener-c792e2d4f0faf7086352849794012c79-1] regionserver.HStore(310): Store=c792e2d4f0faf7086352849794012c79/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=258 2023-02-11 10:00:51,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=258, state=SUCCESS; OpenRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 181 msec 2023-02-11 10:00:51,976 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,977 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:51,977 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf 2023-02-11 10:00:51,977 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, ASSIGN in 336 msec 2023-02-11 10:00:51,978 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1864ff4284b6d01e4d7d9b8e96f54a40 columnFamilyName cf 2023-02-11 10:00:51,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,979 DEBUG [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa 2023-02-11 10:00:51,979 INFO [StoreOpener-2dba5f8399997a147e997c4d38161234-1] regionserver.HStore(310): Store=2dba5f8399997a147e997c4d38161234/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,980 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:51,982 DEBUG [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(539): loaded hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765 2023-02-11 10:00:51,982 INFO [StoreOpener-1864ff4284b6d01e4d7d9b8e96f54a40-1] regionserver.HStore(310): Store=1864ff4284b6d01e4d7d9b8e96f54a40/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 10:00:51,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c792e2d4f0faf7086352849794012c79; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72767655, jitterRate=0.08432255685329437}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c792e2d4f0faf7086352849794012c79: Running coprocessor pre-open hook at 1676109651967Writing region info on filesystem at 1676109651967Initializing all the Stores at 1676109651968 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651968Cleaning up temporary data from old regions at 1676109651978 (+10 ms)Cleaning up detritus from prior splits at 1676109651979 (+1 ms)Running coprocessor post-open hooks at 1676109651982 (+3 ms)Region opened successfully at 1676109651982 2023-02-11 10:00:51,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79., pid=265, masterSystemTime=1676109651947 2023-02-11 10:00:51,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:51,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:51,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:51,985 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:51,985 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109651985"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651985"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651985"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651985"}]},"ts":"1676109651985"} 2023-02-11 10:00:51,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2dba5f8399997a147e997c4d38161234; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65240035, jitterRate=-0.02784772217273712}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2dba5f8399997a147e997c4d38161234: Running coprocessor pre-open hook at 1676109651965Writing region info on filesystem at 1676109651965Initializing all the Stores at 1676109651966 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651966Cleaning up temporary data from old regions at 1676109651982 (+16 ms)Cleaning up detritus from prior splits at 1676109651982Running coprocessor post-open hooks at 1676109651985 (+3 ms)Region opened successfully at 1676109651985 2023-02-11 10:00:51,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234., pid=266, masterSystemTime=1676109651948 2023-02-11 10:00:51,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:51,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:51,987 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=257 2023-02-11 10:00:51,987 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:51,987 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=257, state=SUCCESS; OpenRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 191 msec 2023-02-11 10:00:51,988 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109651987"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651987"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651987"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651987"}]},"ts":"1676109651987"} 2023-02-11 10:00:51,988 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, ASSIGN in 347 msec 2023-02-11 10:00:51,990 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-11 10:00:51,990 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 193 msec 2023-02-11 10:00:51,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, ASSIGN in 350 msec 2023-02-11 10:00:51,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:51,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1864ff4284b6d01e4d7d9b8e96f54a40; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65550257, jitterRate=-0.023225054144859314}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 10:00:51,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Running coprocessor pre-open hook at 1676109651973Writing region info on filesystem at 1676109651973Initializing all the Stores at 1676109651974 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676109651974Cleaning up temporary data from old regions at 1676109651984 (+10 ms)Cleaning up detritus from prior splits at 1676109651985 (+1 ms)Running coprocessor post-open hooks at 1676109651993 (+8 ms)Region opened successfully at 1676109651993 2023-02-11 10:00:51,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40., pid=267, masterSystemTime=1676109651947 2023-02-11 10:00:51,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:51,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:51,995 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:51,995 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109651995"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676109651995"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676109651995"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676109651995"}]},"ts":"1676109651995"} 2023-02-11 10:00:51,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=262 2023-02-11 10:00:51,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=262, state=SUCCESS; OpenRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 199 msec 2023-02-11 10:00:51,999 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=262, resume processing ppid=256 2023-02-11 10:00:51,999 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, ASSIGN in 358 msec 2023-02-11 10:00:52,000 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109652000"}]},"ts":"1676109652000"} 2023-02-11 10:00:52,001 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=ENABLED in hbase:meta 2023-02-11 10:00:52,003 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676109647730' was successfully enabled. 2023-02-11 10:00:52,007 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676109647730 in 372 msec 2023-02-11 10:00:52,358 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-11 10:00:52,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-11 10:00:52,389 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 256 completed 2023-02-11 10:00:52,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41567] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35443] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41193] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. with WAL disabled. Data may be lost in the event of a crash. 2023-02-11 10:00:52,400 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676109647730 2023-02-11 10:00:52,400 DEBUG [Listener at localhost/42305] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:52,400 INFO [Listener at localhost/42305] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-11 10:00:52,507 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-11 10:00:52,614 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-11 10:00:52,674 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-11 10:00:52,724 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-11 10:00:52,817 DEBUG [Listener at localhost/42305] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-11 10:00:53,271 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1913e027 to 127.0.0.1:64863 2023-02-11 10:00:53,271 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:53,885 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 2c0bfd25d86241e7822bae53c725c765 in the real (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) or archive (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) directory for the primary table. 2023-02-11 10:00:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 2c0bfd25d86241e7822bae53c725c765 in the real (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) or archive (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 10:00:53,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] ipc.CallRunner(144): callId: 366 service: MasterService methodName: RestoreSnapshot size: 114 connection: 172.31.14.131:47014 deadline: 1676109713869, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 2c0bfd25d86241e7822bae53c725c765 in the real (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) or archive (hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/2c0bfd25d86241e7822bae53c725c765) directory for the primary table. 2023-02-11 10:00:53,911 INFO [Listener at localhost/42305] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676109647730 2023-02-11 10:00:53,911 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676109647730 2023-02-11 10:00:53,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:53,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 10:00:53,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109653914"}]},"ts":"1676109653914"} 2023-02-11 10:00:53,915 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLING in hbase:meta 2023-02-11 10:00:53,918 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676109647730 to state=DISABLING 2023-02-11 10:00:53,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN}] 2023-02-11 10:00:53,919 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN 2023-02-11 10:00:53,919 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN 2023-02-11 10:00:53,919 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN 2023-02-11 10:00:53,919 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN 2023-02-11 10:00:53,919 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN 2023-02-11 10:00:53,919 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:53,920 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109653919"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653919"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653919"}]},"ts":"1676109653919"} 2023-02-11 10:00:53,920 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:53,920 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:53,920 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:53,920 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109653920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653920"}]},"ts":"1676109653920"} 2023-02-11 10:00:53,920 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109653920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653920"}]},"ts":"1676109653920"} 2023-02-11 10:00:53,920 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:53,920 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109653920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653920"}]},"ts":"1676109653920"} 2023-02-11 10:00:53,920 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109653920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653920"}]},"ts":"1676109653920"} 2023-02-11 10:00:53,921 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:53,921 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327}] 2023-02-11 10:00:53,922 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=274, state=RUNNABLE; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:53,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=276, state=RUNNABLE; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:53,923 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN 2023-02-11 10:00:53,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=272, state=RUNNABLE; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244}] 2023-02-11 10:00:53,924 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:53,924 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109653923"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676109653923"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676109653923"}]},"ts":"1676109653923"} 2023-02-11 10:00:53,925 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293}] 2023-02-11 10:00:54,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:54,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 5b412f238605ea2c2123a33e84ce883e, disabling compactions & flushes 2023-02-11 10:00:54,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:54,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:54,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. after waiting 0 ms 2023-02-11 10:00:54,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:54,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 5b412f238605ea2c2123a33e84ce883e 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-11 10:00:54,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:54,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 613f74e8c853d032f33c9b749dc5b78d, disabling compactions & flushes 2023-02-11 10:00:54,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. after waiting 0 ms 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:54,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 613f74e8c853d032f33c9b749dc5b78d 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-11 10:00:54,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 4f895b4386bc67b999d38d2505ce37ca, disabling compactions & flushes 2023-02-11 10:00:54,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. after waiting 0 ms 2023-02-11 10:00:54,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:54,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 4f895b4386bc67b999d38d2505ce37ca 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-11 10:00:54,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302112582cbab4bd642a2b7d46737039f47ed to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c202302112582cbab4bd642a2b7d46737039f47ed 2023-02-11 10:00:54,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230211929ada57177b4f039258eb961a7482d9 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211929ada57177b4f039258eb961a7482d9 2023-02-11 10:00:54,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302116df1da04ba674bf98408909ac51fc328 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302116df1da04ba674bf98408909ac51fc328 2023-02-11 10:00:54,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/.tmp/cf/2ba20b0e37574bd0807b7fde88329499 2023-02-11 10:00:54,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/.tmp/cf/be4fec0a42934ad1a1d4c9daa28306e6 2023-02-11 10:00:54,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/.tmp/cf/e4321b50af8c45c09eda324bffa55f71 2023-02-11 10:00:54,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/.tmp/cf/2ba20b0e37574bd0807b7fde88329499 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/2ba20b0e37574bd0807b7fde88329499 2023-02-11 10:00:54,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/.tmp/cf/be4fec0a42934ad1a1d4c9daa28306e6 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/be4fec0a42934ad1a1d4c9daa28306e6 2023-02-11 10:00:54,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/.tmp/cf/e4321b50af8c45c09eda324bffa55f71 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/e4321b50af8c45c09eda324bffa55f71 2023-02-11 10:00:54,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/2ba20b0e37574bd0807b7fde88329499, entries=29, sequenceid=15, filesize=9.9 K 2023-02-11 10:00:54,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/e4321b50af8c45c09eda324bffa55f71, entries=24, sequenceid=15, filesize=9.0 K 2023-02-11 10:00:54,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/be4fec0a42934ad1a1d4c9daa28306e6, entries=30, sequenceid=15, filesize=10.1 K 2023-02-11 10:00:54,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 613f74e8c853d032f33c9b749dc5b78d in 37ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5b412f238605ea2c2123a33e84ce883e in 41ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 4f895b4386bc67b999d38d2505ce37ca in 39ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. 2023-02-11 10:00:54,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. 2023-02-11 10:00:54,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 4f895b4386bc67b999d38d2505ce37ca: Waiting for close lock at 1676109654075Running coprocessor pre-close hooks at 1676109654075Disabling compacts and flushes for region at 1676109654075Disabling writes for close at 1676109654075Obtaining lock to block concurrent updates at 1676109654075Preparing flush snapshotting stores in 4f895b4386bc67b999d38d2505ce37ca at 1676109654075Finished memstore snapshotting testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676109654075Flushing stores of testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca. at 1676109654077 (+2 ms)Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: creating writer at 1676109654078 (+1 ms)Flushing mob file 4f895b4386bc67b999d38d2505ce37ca/cf: appending metadata at 1676109654081 (+3 ms)Flushing mob file 4f895b4386bc67b999d38d2505ce37ca/cf: closing flushed file at 1676109654081Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: appending metadata at 1676109654097 (+16 ms)Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: closing flushed file at 1676109654097Flushing 4f895b4386bc67b999d38d2505ce37ca/cf: reopening flushed file at 1676109654108 (+11 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 4f895b4386bc67b999d38d2505ce37ca in 39ms, sequenceid=15, compaction requested=false at 1676109654114 (+6 ms)Writing region close event to WAL at 1676109654116 (+2 ms)Running coprocessor post-close hooks at 1676109654119 (+3 ms)Closed at 1676109654120 (+1 ms) 2023-02-11 10:00:54,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 613f74e8c853d032f33c9b749dc5b78d: Waiting for close lock at 1676109654073Running coprocessor pre-close hooks at 1676109654073Disabling compacts and flushes for region at 1676109654073Disabling writes for close at 1676109654075 (+2 ms)Obtaining lock to block concurrent updates at 1676109654075Preparing flush snapshotting stores in 613f74e8c853d032f33c9b749dc5b78d at 1676109654075Finished memstore snapshotting testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676109654075Flushing stores of testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d. at 1676109654075Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: creating writer at 1676109654076 (+1 ms)Flushing mob file 613f74e8c853d032f33c9b749dc5b78d/cf: appending metadata at 1676109654080 (+4 ms)Flushing mob file 613f74e8c853d032f33c9b749dc5b78d/cf: closing flushed file at 1676109654080Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: appending metadata at 1676109654099 (+19 ms)Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: closing flushed file at 1676109654099Flushing 613f74e8c853d032f33c9b749dc5b78d/cf: reopening flushed file at 1676109654109 (+10 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for 613f74e8c853d032f33c9b749dc5b78d in 37ms, sequenceid=15, compaction requested=false at 1676109654112 (+3 ms)Writing region close event to WAL at 1676109654116 (+4 ms)Running coprocessor post-close hooks at 1676109654120 (+4 ms)Closed at 1676109654120 2023-02-11 10:00:54,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. 2023-02-11 10:00:54,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 5b412f238605ea2c2123a33e84ce883e: Waiting for close lock at 1676109654073Running coprocessor pre-close hooks at 1676109654073Disabling compacts and flushes for region at 1676109654073Disabling writes for close at 1676109654073Obtaining lock to block concurrent updates at 1676109654073Preparing flush snapshotting stores in 5b412f238605ea2c2123a33e84ce883e at 1676109654073Finished memstore snapshotting testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676109654073Flushing stores of testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e. at 1676109654073Flushing 5b412f238605ea2c2123a33e84ce883e/cf: creating writer at 1676109654074 (+1 ms)Flushing mob file 5b412f238605ea2c2123a33e84ce883e/cf: appending metadata at 1676109654080 (+6 ms)Flushing mob file 5b412f238605ea2c2123a33e84ce883e/cf: closing flushed file at 1676109654080Flushing 5b412f238605ea2c2123a33e84ce883e/cf: appending metadata at 1676109654097 (+17 ms)Flushing 5b412f238605ea2c2123a33e84ce883e/cf: closing flushed file at 1676109654097Flushing 5b412f238605ea2c2123a33e84ce883e/cf: reopening flushed file at 1676109654109 (+12 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5b412f238605ea2c2123a33e84ce883e in 41ms, sequenceid=15, compaction requested=false at 1676109654114 (+5 ms)Writing region close event to WAL at 1676109654116 (+2 ms)Running coprocessor post-close hooks at 1676109654120 (+4 ms)Closed at 1676109654121 (+1 ms) 2023-02-11 10:00:54,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:54,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:54,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1864ff4284b6d01e4d7d9b8e96f54a40, disabling compactions & flushes 2023-02-11 10:00:54,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:54,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:54,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. after waiting 0 ms 2023-02-11 10:00:54,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:54,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1864ff4284b6d01e4d7d9b8e96f54a40 1/1 column families, dataSize=23.42 KB heapSize=50.59 KB 2023-02-11 10:00:54,122 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=4f895b4386bc67b999d38d2505ce37ca, regionState=CLOSED 2023-02-11 10:00:54,123 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109654122"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654122"}]},"ts":"1676109654122"} 2023-02-11 10:00:54,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:54,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:54,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c792e2d4f0faf7086352849794012c79, disabling compactions & flushes 2023-02-11 10:00:54,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:54,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:54,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. after waiting 0 ms 2023-02-11 10:00:54,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:54,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c792e2d4f0faf7086352849794012c79 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-11 10:00:54,125 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=613f74e8c853d032f33c9b749dc5b78d, regionState=CLOSED 2023-02-11 10:00:54,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109654124"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654124"}]},"ts":"1676109654124"} 2023-02-11 10:00:54,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:54,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:54,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2dba5f8399997a147e997c4d38161234, disabling compactions & flushes 2023-02-11 10:00:54,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:54,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:54,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. after waiting 0 ms 2023-02-11 10:00:54,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:54,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2dba5f8399997a147e997c4d38161234 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-11 10:00:54,126 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=5b412f238605ea2c2123a33e84ce883e, regionState=CLOSED 2023-02-11 10:00:54,126 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109654126"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654126"}]},"ts":"1676109654126"} 2023-02-11 10:00:54,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=272 2023-02-11 10:00:54,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=272, state=SUCCESS; CloseRegionProcedure 4f895b4386bc67b999d38d2505ce37ca, server=jenkins-hbase4.apache.org,41193,1676109614244 in 203 msec 2023-02-11 10:00:54,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=274 2023-02-11 10:00:54,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=274, state=SUCCESS; CloseRegionProcedure 613f74e8c853d032f33c9b749dc5b78d, server=jenkins-hbase4.apache.org,41567,1676109614293 in 205 msec 2023-02-11 10:00:54,132 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=4f895b4386bc67b999d38d2505ce37ca, UNASSIGN in 211 msec 2023-02-11 10:00:54,133 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-11 10:00:54,133 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 5b412f238605ea2c2123a33e84ce883e, server=jenkins-hbase4.apache.org,35443,1676109614327 in 207 msec 2023-02-11 10:00:54,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=613f74e8c853d032f33c9b749dc5b78d, UNASSIGN in 212 msec 2023-02-11 10:00:54,134 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=5b412f238605ea2c2123a33e84ce883e, UNASSIGN in 215 msec 2023-02-11 10:00:54,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302117723db00570641b0b04d177e51a26d9f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302117723db00570641b0b04d177e51a26d9f 2023-02-11 10:00:54,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302115f34ca473fa842a9a2e6780be80b77e8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302115f34ca473fa842a9a2e6780be80b77e8 2023-02-11 10:00:54,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230211db46ca5937504852ae3a974bbc1933e5 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e20230211db46ca5937504852ae3a974bbc1933e5 2023-02-11 10:00:54,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 10:00:54,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/.tmp/cf/3d4a85a1629c436b94ac2a8474dcf5a6 2023-02-11 10:00:54,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/.tmp/cf/3a096d4efe954529b80d79d8da4fdcbb 2023-02-11 10:00:54,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/.tmp/cf/781b704292744a2b85c5a32351d607b8 2023-02-11 10:00:54,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/.tmp/cf/3a096d4efe954529b80d79d8da4fdcbb as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/3a096d4efe954529b80d79d8da4fdcbb 2023-02-11 10:00:54,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/.tmp/cf/3d4a85a1629c436b94ac2a8474dcf5a6 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/3d4a85a1629c436b94ac2a8474dcf5a6 2023-02-11 10:00:54,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/.tmp/cf/781b704292744a2b85c5a32351d607b8 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/781b704292744a2b85c5a32351d607b8 2023-02-11 10:00:54,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/3a096d4efe954529b80d79d8da4fdcbb, entries=358, sequenceid=15, filesize=66.6 K 2023-02-11 10:00:54,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/3d4a85a1629c436b94ac2a8474dcf5a6, entries=33, sequenceid=15, filesize=10.6 K 2023-02-11 10:00:54,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.42 KB/23982, heapSize ~50.58 KB/51792, currentSize=0 B/0 for 1864ff4284b6d01e4d7d9b8e96f54a40 in 54ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 2dba5f8399997a147e997c4d38161234 in 51ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/781b704292744a2b85c5a32351d607b8, entries=26, sequenceid=15, filesize=9.4 K 2023-02-11 10:00:54,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for c792e2d4f0faf7086352849794012c79 in 54ms, sequenceid=15, compaction requested=false 2023-02-11 10:00:54,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. 2023-02-11 10:00:54,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1864ff4284b6d01e4d7d9b8e96f54a40: Waiting for close lock at 1676109654122Running coprocessor pre-close hooks at 1676109654122Disabling compacts and flushes for region at 1676109654122Disabling writes for close at 1676109654122Obtaining lock to block concurrent updates at 1676109654122Preparing flush snapshotting stores in 1864ff4284b6d01e4d7d9b8e96f54a40 at 1676109654122Finished memstore snapshotting testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40., syncing WAL and waiting on mvcc, flushsize=dataSize=23982, getHeapSize=51792, getOffHeapSize=0, getCellsCount=358 at 1676109654123 (+1 ms)Flushing stores of testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40. at 1676109654123Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: creating writer at 1676109654124 (+1 ms)Flushing mob file 1864ff4284b6d01e4d7d9b8e96f54a40/cf: appending metadata at 1676109654131 (+7 ms)Flushing mob file 1864ff4284b6d01e4d7d9b8e96f54a40/cf: closing flushed file at 1676109654131Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: appending metadata at 1676109654150 (+19 ms)Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: closing flushed file at 1676109654150Flushing 1864ff4284b6d01e4d7d9b8e96f54a40/cf: reopening flushed file at 1676109654172 (+22 ms)Finished flush of dataSize ~23.42 KB/23982, heapSize ~50.58 KB/51792, currentSize=0 B/0 for 1864ff4284b6d01e4d7d9b8e96f54a40 in 54ms, sequenceid=15, compaction requested=false at 1676109654176 (+4 ms)Writing region close event to WAL at 1676109654177 (+1 ms)Running coprocessor post-close hooks at 1676109654180 (+3 ms)Closed at 1676109654181 (+1 ms) 2023-02-11 10:00:54,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. 2023-02-11 10:00:54,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2dba5f8399997a147e997c4d38161234: Waiting for close lock at 1676109654125Running coprocessor pre-close hooks at 1676109654125Disabling compacts and flushes for region at 1676109654125Disabling writes for close at 1676109654125Obtaining lock to block concurrent updates at 1676109654125Preparing flush snapshotting stores in 2dba5f8399997a147e997c4d38161234 at 1676109654125Finished memstore snapshotting testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676109654125Flushing stores of testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234. at 1676109654127 (+2 ms)Flushing 2dba5f8399997a147e997c4d38161234/cf: creating writer at 1676109654127Flushing mob file 2dba5f8399997a147e997c4d38161234/cf: appending metadata at 1676109654131 (+4 ms)Flushing mob file 2dba5f8399997a147e997c4d38161234/cf: closing flushed file at 1676109654131Flushing 2dba5f8399997a147e997c4d38161234/cf: appending metadata at 1676109654150 (+19 ms)Flushing 2dba5f8399997a147e997c4d38161234/cf: closing flushed file at 1676109654150Flushing 2dba5f8399997a147e997c4d38161234/cf: reopening flushed file at 1676109654173 (+23 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 2dba5f8399997a147e997c4d38161234 in 51ms, sequenceid=15, compaction requested=false at 1676109654176 (+3 ms)Writing region close event to WAL at 1676109654177 (+1 ms)Running coprocessor post-close hooks at 1676109654181 (+4 ms)Closed at 1676109654182 (+1 ms) 2023-02-11 10:00:54,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:54,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-11 10:00:54,183 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=1864ff4284b6d01e4d7d9b8e96f54a40, regionState=CLOSED 2023-02-11 10:00:54,183 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109654183"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654183"}]},"ts":"1676109654183"} 2023-02-11 10:00:54,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:54,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. 2023-02-11 10:00:54,184 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=2dba5f8399997a147e997c4d38161234, regionState=CLOSED 2023-02-11 10:00:54,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c792e2d4f0faf7086352849794012c79: Waiting for close lock at 1676109654124Running coprocessor pre-close hooks at 1676109654124Disabling compacts and flushes for region at 1676109654124Disabling writes for close at 1676109654124Obtaining lock to block concurrent updates at 1676109654124Preparing flush snapshotting stores in c792e2d4f0faf7086352849794012c79 at 1676109654124Finished memstore snapshotting testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676109654125 (+1 ms)Flushing stores of testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79. at 1676109654125Flushing c792e2d4f0faf7086352849794012c79/cf: creating writer at 1676109654126 (+1 ms)Flushing mob file c792e2d4f0faf7086352849794012c79/cf: appending metadata at 1676109654135 (+9 ms)Flushing mob file c792e2d4f0faf7086352849794012c79/cf: closing flushed file at 1676109654135Flushing c792e2d4f0faf7086352849794012c79/cf: appending metadata at 1676109654152 (+17 ms)Flushing c792e2d4f0faf7086352849794012c79/cf: closing flushed file at 1676109654152Flushing c792e2d4f0faf7086352849794012c79/cf: reopening flushed file at 1676109654175 (+23 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for c792e2d4f0faf7086352849794012c79 in 54ms, sequenceid=15, compaction requested=false at 1676109654178 (+3 ms)Writing region close event to WAL at 1676109654179 (+1 ms)Running coprocessor post-close hooks at 1676109654183 (+4 ms)Closed at 1676109654184 (+1 ms) 2023-02-11 10:00:54,184 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676109654183"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654183"}]},"ts":"1676109654183"} 2023-02-11 10:00:54,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:54,186 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=c792e2d4f0faf7086352849794012c79, regionState=CLOSED 2023-02-11 10:00:54,186 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676109654186"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676109654186"}]},"ts":"1676109654186"} 2023-02-11 10:00:54,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=276 2023-02-11 10:00:54,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=276, state=SUCCESS; CloseRegionProcedure 1864ff4284b6d01e4d7d9b8e96f54a40, server=jenkins-hbase4.apache.org,41193,1676109614244 in 262 msec 2023-02-11 10:00:54,187 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-02-11 10:00:54,187 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure 2dba5f8399997a147e997c4d38161234, server=jenkins-hbase4.apache.org,35443,1676109614327 in 265 msec 2023-02-11 10:00:54,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=1864ff4284b6d01e4d7d9b8e96f54a40, UNASSIGN in 269 msec 2023-02-11 10:00:54,188 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=2dba5f8399997a147e997c4d38161234, UNASSIGN in 269 msec 2023-02-11 10:00:54,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-11 10:00:54,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure c792e2d4f0faf7086352849794012c79, server=jenkins-hbase4.apache.org,41567,1676109614293 in 262 msec 2023-02-11 10:00:54,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-02-11 10:00:54,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676109647730, region=c792e2d4f0faf7086352849794012c79, UNASSIGN in 271 msec 2023-02-11 10:00:54,190 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676109654190"}]},"ts":"1676109654190"} 2023-02-11 10:00:54,191 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676109647730, state=DISABLED in hbase:meta 2023-02-11 10:00:54,193 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676109647730 to state=DISABLED 2023-02-11 10:00:54,198 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676109647730 in 287 msec 2023-02-11 10:00:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-11 10:00:54,667 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 270 completed 2023-02-11 10:00:54,667 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,670 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 10:00:54,671 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:54,681 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:54,684 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits] 2023-02-11 10:00:54,685 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits] 2023-02-11 10:00:54,686 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits] 2023-02-11 10:00:54,687 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits] 2023-02-11 10:00:54,687 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits] 2023-02-11 10:00:54,687 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf, FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits] 2023-02-11 10:00:54,692 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/3a096d4efe954529b80d79d8da4fdcbb to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/cf/3a096d4efe954529b80d79d8da4fdcbb 2023-02-11 10:00:54,694 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/2ba20b0e37574bd0807b7fde88329499 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/2ba20b0e37574bd0807b7fde88329499 2023-02-11 10:00:54,695 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/781b704292744a2b85c5a32351d607b8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/781b704292744a2b85c5a32351d607b8 2023-02-11 10:00:54,696 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/89bec9e0b6e2406e9cb1c17ab313fc06 2023-02-11 10:00:54,697 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/cf/f6d039d021da4ea7a0e9bd3d87936f52 2023-02-11 10:00:54,697 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/cf/878eb54fdc21427ca36fd5e2f4a7890f 2023-02-11 10:00:54,698 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/3d4a85a1629c436b94ac2a8474dcf5a6 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/3d4a85a1629c436b94ac2a8474dcf5a6 2023-02-11 10:00:54,698 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/6cbc83dae62c40d18fd9754e3619dc2e 2023-02-11 10:00:54,700 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/be4fec0a42934ad1a1d4c9daa28306e6 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/cf/be4fec0a42934ad1a1d4c9daa28306e6 2023-02-11 10:00:54,701 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40/recovered.edits/18.seqid 2023-02-11 10:00:54,701 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/e4321b50af8c45c09eda324bffa55f71 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/cf/e4321b50af8c45c09eda324bffa55f71 2023-02-11 10:00:54,702 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/cf/6b78918e0a344151b41e0b5d928f6afa 2023-02-11 10:00:54,702 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/1864ff4284b6d01e4d7d9b8e96f54a40 2023-02-11 10:00:54,704 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca/recovered.edits/18.seqid 2023-02-11 10:00:54,705 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/4f895b4386bc67b999d38d2505ce37ca 2023-02-11 10:00:54,705 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79/recovered.edits/18.seqid 2023-02-11 10:00:54,706 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/c792e2d4f0faf7086352849794012c79 2023-02-11 10:00:54,706 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d/recovered.edits/18.seqid 2023-02-11 10:00:54,706 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e/recovered.edits/18.seqid 2023-02-11 10:00:54,707 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/613f74e8c853d032f33c9b749dc5b78d 2023-02-11 10:00:54,707 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/5b412f238605ea2c2123a33e84ce883e 2023-02-11 10:00:54,707 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/18.seqid to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234/recovered.edits/18.seqid 2023-02-11 10:00:54,708 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/.tmp/data/default/testCorruptedSnapshot-1676109647730/2dba5f8399997a147e997c4d38161234 2023-02-11 10:00:54,708 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676109647730 regions 2023-02-11 10:00:54,708 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00 2023-02-11 10:00:54,709 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf] 2023-02-11 10:00:54,712 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c2023021123d1fa251e824303805e0d8285eef5b8 2023-02-11 10:00:54,713 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c202302112582cbab4bd642a2b7d46737039f47ed to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/a87ff679a2f3e71d9181a67b7542122c202302112582cbab4bd642a2b7d46737039f47ed 2023-02-11 10:00:54,714 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211929ada57177b4f039258eb961a7482d9 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211929ada57177b4f039258eb961a7482d9 2023-02-11 10:00:54,715 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c4ca4238a0b923820dcc509a6f75849b20230211cf65d67c7692491b81eabefa6269c111 2023-02-11 10:00:54,717 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302114ee8f8fb57c84fda8a13f2ea33e63e28 2023-02-11 10:00:54,718 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302117723db00570641b0b04d177e51a26d9f to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/c81e728d9d4c2f636f067f89cc14862c202302117723db00570641b0b04d177e51a26d9f 2023-02-11 10:00:54,719 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e202302117bbf0de97d804011bb2cb0f0f4af6703 2023-02-11 10:00:54,720 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e20230211db46ca5937504852ae3a974bbc1933e5 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/d41d8cd98f00b204e9800998ecf8427e20230211db46ca5937504852ae3a974bbc1933e5 2023-02-11 10:00:54,722 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302112e8a4aa152c84346bc9735f3a75526ac 2023-02-11 10:00:54,723 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302115f34ca473fa842a9a2e6780be80b77e8 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/e4da3b7fbbce2345d7772b0674a318d5202302115f34ca473fa842a9a2e6780be80b77e8 2023-02-11 10:00:54,725 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302116df1da04ba674bf98408909ac51fc328 to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302116df1da04ba674bf98408909ac51fc328 2023-02-11 10:00:54,726 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa to hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/archive/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230211e06b317a310648e1bb04089cc70d3eaa 2023-02-11 10:00:54,727 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/mobdir/data/default/testCorruptedSnapshot-1676109647730/8310531b59856ff2d2ebb8de1bb26c00 2023-02-11 10:00:54,729 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,732 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676109647730 from hbase:meta 2023-02-11 10:00:54,742 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676109647730' descriptor. 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676109647730' from region states. 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676109654744"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,746 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-11 10:00:54,746 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c792e2d4f0faf7086352849794012c79, NAME => 'testCorruptedSnapshot-1676109647730,,1676109647734.c792e2d4f0faf7086352849794012c79.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 4f895b4386bc67b999d38d2505ce37ca, NAME => 'testCorruptedSnapshot-1676109647730,1,1676109647734.4f895b4386bc67b999d38d2505ce37ca.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 2dba5f8399997a147e997c4d38161234, NAME => 'testCorruptedSnapshot-1676109647730,2,1676109647734.2dba5f8399997a147e997c4d38161234.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 613f74e8c853d032f33c9b749dc5b78d, NAME => 'testCorruptedSnapshot-1676109647730,3,1676109647734.613f74e8c853d032f33c9b749dc5b78d.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 5b412f238605ea2c2123a33e84ce883e, NAME => 'testCorruptedSnapshot-1676109647730,4,1676109647734.5b412f238605ea2c2123a33e84ce883e.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 1864ff4284b6d01e4d7d9b8e96f54a40, NAME => 'testCorruptedSnapshot-1676109647730,5,1676109647734.1864ff4284b6d01e4d7d9b8e96f54a40.', STARTKEY => '5', ENDKEY => ''}] 2023-02-11 10:00:54,746 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676109647730' as deleted. 2023-02-11 10:00:54,747 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676109647730","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676109654747"}]},"ts":"9223372036854775807"} 2023-02-11 10:00:54,748 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676109647730 state from META 2023-02-11 10:00:54,750 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 2023-02-11 10:00:54,751 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676109647730 in 82 msec 2023-02-11 10:00:54,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-11 10:00:54,923 INFO [Listener at localhost/42305] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676109647730, procId: 283 completed 2023-02-11 10:00:54,929 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676109647730" 2023-02-11 10:00:54,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676109647730 2023-02-11 10:00:54,932 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676109647730" 2023-02-11 10:00:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42723] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676109647730 2023-02-11 10:00:54,954 INFO [Listener at localhost/42305] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=419 (was 407) Potentially hanging thread: hconnection-0x6de71207-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1417049990_15 at /127.0.0.1:51502 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_972808233_15 at /127.0.0.1:36662 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-617725852_15 at /127.0.0.1:36502 [Waiting for operation #551] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1291327068_15 at /127.0.0.1:46404 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1735626783_15 at /127.0.0.1:36998 [Waiting for operation #266] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:52366 [Waiting for operation #734] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1417049990_15 at /127.0.0.1:36990 [Waiting for operation #261] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:46780 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1417049990_15 at /127.0.0.1:52376 [Waiting for operation #736] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3d381091-shared-pool-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_972808233_15 at /127.0.0.1:46824 [Waiting for operation #263] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:54484 [Waiting for operation #525] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1130755439_15 at /127.0.0.1:46818 [Waiting for operation #267] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6de71207-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=782 (was 774) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=345 (was 335) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=2106 (was 2329) 2023-02-11 10:00:54,954 INFO [Listener at localhost/42305] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 10:00:54,954 INFO [Listener at localhost/42305] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-11 10:00:54,956 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x297db6e1 to 127.0.0.1:64863 2023-02-11 10:00:54,956 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,957 DEBUG [Listener at localhost/42305] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 10:00:54,957 DEBUG [Listener at localhost/42305] util.JVMClusterUtil(257): Found active master hash=239353085, stopped=false 2023-02-11 10:00:54,957 INFO [Listener at localhost/42305] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:54,963 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:54,963 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:54,963 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:54,963 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 10:00:54,963 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:54,963 INFO [Listener at localhost/42305] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 10:00:54,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:54,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:54,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:54,964 DEBUG [Listener at localhost/42305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08a97f85 to 127.0.0.1:64863 2023-02-11 10:00:54,964 DEBUG [Listener at localhost/42305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 10:00:54,965 INFO [Listener at localhost/42305] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41193,1676109614244' ***** 2023-02-11 10:00:54,965 INFO [Listener at localhost/42305] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 10:00:54,965 INFO [Listener at localhost/42305] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41567,1676109614293' ***** 2023-02-11 10:00:54,965 INFO [Listener at localhost/42305] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 10:00:54,965 INFO [RS:0;jenkins-hbase4:41193] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 10:00:54,965 INFO [RS:1;jenkins-hbase4:41567] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 10:00:54,965 INFO [RS:0;jenkins-hbase4:41193] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 10:00:54,966 INFO [RS:0;jenkins-hbase4:41193] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 10:00:54,966 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 10:00:54,966 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 10:00:54,965 INFO [Listener at localhost/42305] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35443,1676109614327' ***** 2023-02-11 10:00:54,966 INFO [Listener at localhost/42305] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 10:00:54,966 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:54,966 INFO [RS:2;jenkins-hbase4:35443] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 10:00:54,965 INFO [RS:1;jenkins-hbase4:41567] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 10:00:54,966 INFO [RS:2;jenkins-hbase4:35443] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 10:00:54,966 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 10:00:54,966 INFO [RS:1;jenkins-hbase4:41567] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 10:00:54,967 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(3304): Received CLOSE for f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:54,967 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:54,967 DEBUG [RS:1;jenkins-hbase4:41567] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a654399 to 127.0.0.1:64863 2023-02-11 10:00:54,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f5663c8b8537d642d1212d0ff668c8a1, disabling compactions & flushes 2023-02-11 10:00:54,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:54,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:54,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. after waiting 0 ms 2023-02-11 10:00:54,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:54,966 DEBUG [RS:0;jenkins-hbase4:41193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17fbc61c to 127.0.0.1:64863 2023-02-11 10:00:54,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f5663c8b8537d642d1212d0ff668c8a1 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 10:00:54,967 DEBUG [RS:1;jenkins-hbase4:41567] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,968 INFO [RS:1;jenkins-hbase4:41567] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 10:00:54,968 INFO [RS:1;jenkins-hbase4:41567] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 10:00:54,968 INFO [RS:1;jenkins-hbase4:41567] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 10:00:54,968 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 10:00:54,966 INFO [RS:2;jenkins-hbase4:35443] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 10:00:54,969 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:54,969 DEBUG [RS:2;jenkins-hbase4:35443] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e33b76 to 127.0.0.1:64863 2023-02-11 10:00:54,969 DEBUG [RS:2;jenkins-hbase4:35443] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,969 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35443,1676109614327; all regions closed. 2023-02-11 10:00:54,967 DEBUG [RS:0;jenkins-hbase4:41193] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,969 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41193,1676109614244; all regions closed. 2023-02-11 10:00:54,970 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-11 10:00:54,970 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, f5663c8b8537d642d1212d0ff668c8a1=hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1.} 2023-02-11 10:00:54,971 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:54,971 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:54,972 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:54,972 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 10:00:54,972 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 10:00:54,972 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 10:00:54,972 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 10:00:54,972 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 10:00:54,972 DEBUG [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1505): Waiting on 1588230740, f5663c8b8537d642d1212d0ff668c8a1 2023-02-11 10:00:54,973 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-11 10:00:54,973 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-11 10:00:54,973 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-11 10:00:54,983 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-11 10:00:54,983 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-11 10:00:54,990 DEBUG [RS:0;jenkins-hbase4:41193] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs 2023-02-11 10:00:54,990 INFO [RS:0;jenkins-hbase4:41193] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41193%2C1676109614244:(num 1676109616107) 2023-02-11 10:00:54,990 DEBUG [RS:0;jenkins-hbase4:41193] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,990 INFO [RS:0;jenkins-hbase4:41193] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:54,991 DEBUG [RS:2;jenkins-hbase4:35443] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs 2023-02-11 10:00:54,991 INFO [RS:2;jenkins-hbase4:35443] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35443%2C1676109614327:(num 1676109616108) 2023-02-11 10:00:54,991 DEBUG [RS:2;jenkins-hbase4:35443] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:54,991 INFO [RS:2;jenkins-hbase4:35443] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:54,996 INFO [RS:0;jenkins-hbase4:41193] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 10:00:54,996 INFO [RS:0;jenkins-hbase4:41193] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 10:00:54,996 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 10:00:54,996 INFO [RS:0;jenkins-hbase4:41193] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 10:00:54,996 INFO [RS:0;jenkins-hbase4:41193] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 10:00:54,996 INFO [RS:2;jenkins-hbase4:35443] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 10:00:54,997 INFO [RS:2;jenkins-hbase4:35443] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 10:00:54,998 INFO [RS:2;jenkins-hbase4:35443] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 10:00:54,998 INFO [RS:2;jenkins-hbase4:35443] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 10:00:54,997 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 10:00:55,005 INFO [RS:0;jenkins-hbase4:41193] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41193 2023-02-11 10:00:55,005 INFO [RS:2;jenkins-hbase4:35443] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35443 2023-02-11 10:00:55,023 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/info/11ec43c7e02946818b9b5893de2e0411 2023-02-11 10:00:55,027 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 11ec43c7e02946818b9b5893de2e0411 2023-02-11 10:00:55,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/.tmp/info/c49c843bb4864cc48714e1e5b61aa63d 2023-02-11 10:00:55,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/.tmp/info/c49c843bb4864cc48714e1e5b61aa63d as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/info/c49c843bb4864cc48714e1e5b61aa63d 2023-02-11 10:00:55,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/info/c49c843bb4864cc48714e1e5b61aa63d, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 10:00:55,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for f5663c8b8537d642d1212d0ff668c8a1 in 79ms, sequenceid=6, compaction requested=false 2023-02-11 10:00:55,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/namespace/f5663c8b8537d642d1212d0ff668c8a1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 10:00:55,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:55,052 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/rep_barrier/49f698496d3a48c7973ab032a22b3b4b 2023-02-11 10:00:55,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f5663c8b8537d642d1212d0ff668c8a1: Waiting for close lock at 1676109654967Running coprocessor pre-close hooks at 1676109654967Disabling compacts and flushes for region at 1676109654967Disabling writes for close at 1676109654967Obtaining lock to block concurrent updates at 1676109654967Preparing flush snapshotting stores in f5663c8b8537d642d1212d0ff668c8a1 at 1676109654967Finished memstore snapshotting hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676109654968 (+1 ms)Flushing stores of hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. at 1676109654969 (+1 ms)Flushing f5663c8b8537d642d1212d0ff668c8a1/info: creating writer at 1676109654969Flushing f5663c8b8537d642d1212d0ff668c8a1/info: appending metadata at 1676109654974 (+5 ms)Flushing f5663c8b8537d642d1212d0ff668c8a1/info: closing flushed file at 1676109654975 (+1 ms)Flushing f5663c8b8537d642d1212d0ff668c8a1/info: reopening flushed file at 1676109655042 (+67 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for f5663c8b8537d642d1212d0ff668c8a1 in 79ms, sequenceid=6, compaction requested=false at 1676109655046 (+4 ms)Writing region close event to WAL at 1676109655048 (+2 ms)Running coprocessor post-close hooks at 1676109655051 (+3 ms)Closed at 1676109655051 2023-02-11 10:00:55,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676109616548.f5663c8b8537d642d1212d0ff668c8a1. 2023-02-11 10:00:55,055 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49f698496d3a48c7973ab032a22b3b4b 2023-02-11 10:00:55,077 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:55,077 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:55,077 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:55,077 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:55,077 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2af18af2 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,077 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35443,1676109614327 2023-02-11 10:00:55,078 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:55,078 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:55,079 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7cedfbf4 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,079 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:55,079 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4266d03 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,079 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:55,079 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41193,1676109614244 2023-02-11 10:00:55,079 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6adf82dc rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,085 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/table/ec5b574311cb41eda445ee6b6bff348e 2023-02-11 10:00:55,090 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ec5b574311cb41eda445ee6b6bff348e 2023-02-11 10:00:55,090 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/info/11ec43c7e02946818b9b5893de2e0411 as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/info/11ec43c7e02946818b9b5893de2e0411 2023-02-11 10:00:55,094 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 11ec43c7e02946818b9b5893de2e0411 2023-02-11 10:00:55,094 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/info/11ec43c7e02946818b9b5893de2e0411, entries=64, sequenceid=307, filesize=11.8 K 2023-02-11 10:00:55,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/rep_barrier/49f698496d3a48c7973ab032a22b3b4b as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/rep_barrier/49f698496d3a48c7973ab032a22b3b4b 2023-02-11 10:00:55,099 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49f698496d3a48c7973ab032a22b3b4b 2023-02-11 10:00:55,099 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/rep_barrier/49f698496d3a48c7973ab032a22b3b4b, entries=18, sequenceid=307, filesize=7.0 K 2023-02-11 10:00:55,099 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/.tmp/table/ec5b574311cb41eda445ee6b6bff348e as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/table/ec5b574311cb41eda445ee6b6bff348e 2023-02-11 10:00:55,103 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ec5b574311cb41eda445ee6b6bff348e 2023-02-11 10:00:55,103 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/table/ec5b574311cb41eda445ee6b6bff348e, entries=22, sequenceid=307, filesize=7.0 K 2023-02-11 10:00:55,104 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 132ms, sequenceid=307, compaction requested=false 2023-02-11 10:00:55,109 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-11 10:00:55,110 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 10:00:55,112 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 10:00:55,112 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676109654972Running coprocessor pre-close hooks at 1676109654972Disabling compacts and flushes for region at 1676109654972Disabling writes for close at 1676109654972Obtaining lock to block concurrent updates at 1676109654973 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676109654973Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676109654973Flushing stores of hbase:meta,,1.1588230740 at 1676109654974 (+1 ms)Flushing 1588230740/info: creating writer at 1676109654975 (+1 ms)Flushing 1588230740/info: appending metadata at 1676109654991 (+16 ms)Flushing 1588230740/info: closing flushed file at 1676109654991Flushing 1588230740/rep_barrier: creating writer at 1676109655027 (+36 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676109655030 (+3 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676109655030Flushing 1588230740/table: creating writer at 1676109655055 (+25 ms)Flushing 1588230740/table: appending metadata at 1676109655063 (+8 ms)Flushing 1588230740/table: closing flushed file at 1676109655063Flushing 1588230740/info: reopening flushed file at 1676109655091 (+28 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676109655096 (+5 ms)Flushing 1588230740/table: reopening flushed file at 1676109655100 (+4 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 132ms, sequenceid=307, compaction requested=false at 1676109655104 (+4 ms)Writing region close event to WAL at 1676109655106 (+2 ms)Running coprocessor post-close hooks at 1676109655110 (+4 ms)Closed at 1676109655112 (+2 ms) 2023-02-11 10:00:55,112 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 10:00:55,171 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41193,1676109614244] 2023-02-11 10:00:55,171 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41193,1676109614244; numProcessing=1 2023-02-11 10:00:55,173 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41567,1676109614293; all regions closed. 2023-02-11 10:00:55,179 DEBUG [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs 2023-02-11 10:00:55,179 INFO [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41567%2C1676109614293.meta:.meta(num 1676109616267) 2023-02-11 10:00:55,187 DEBUG [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/oldWALs 2023-02-11 10:00:55,187 INFO [RS:1;jenkins-hbase4:41567] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41567%2C1676109614293:(num 1676109616108) 2023-02-11 10:00:55,187 DEBUG [RS:1;jenkins-hbase4:41567] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:55,187 INFO [RS:1;jenkins-hbase4:41567] regionserver.LeaseManager(133): Closed leases 2023-02-11 10:00:55,187 INFO [RS:1;jenkins-hbase4:41567] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 10:00:55,187 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 10:00:55,188 INFO [RS:1;jenkins-hbase4:41567] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41567 2023-02-11 10:00:55,271 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,272 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@cdc0cbc rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,272 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41193-0x10042d2edb60001, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,272 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@74d77c65 rejected from java.util.concurrent.ThreadPoolExecutor@6ee01b5f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,272 INFO [RS:0;jenkins-hbase4:41193] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41193,1676109614244; zookeeper connection closed. 2023-02-11 10:00:55,289 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1cd7501d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1cd7501d 2023-02-11 10:00:55,289 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41567,1676109614293 2023-02-11 10:00:55,289 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@16af48b1 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,289 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41193,1676109614244 already deleted, retry=false 2023-02-11 10:00:55,290 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41193,1676109614244 expired; onlineServers=2 2023-02-11 10:00:55,290 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35443,1676109614327] 2023-02-11 10:00:55,290 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35443,1676109614327; numProcessing=2 2023-02-11 10:00:55,289 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 10:00:55,291 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35443,1676109614327 already deleted, retry=false 2023-02-11 10:00:55,291 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35443,1676109614327 expired; onlineServers=1 2023-02-11 10:00:55,292 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41567,1676109614293] 2023-02-11 10:00:55,292 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41567,1676109614293; numProcessing=3 2023-02-11 10:00:55,293 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41567,1676109614293 already deleted, retry=false 2023-02-11 10:00:55,293 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41567,1676109614293 expired; onlineServers=0 2023-02-11 10:00:55,293 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42723,1676109612965' ***** 2023-02-11 10:00:55,293 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 10:00:55,293 DEBUG [M:0;jenkins-hbase4:42723] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f7936c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 10:00:55,293 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42723,1676109612965 2023-02-11 10:00:55,293 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42723,1676109612965; all regions closed. 2023-02-11 10:00:55,293 DEBUG [M:0;jenkins-hbase4:42723] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 10:00:55,293 DEBUG [M:0;jenkins-hbase4:42723] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 10:00:55,294 DEBUG [M:0;jenkins-hbase4:42723] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 10:00:55,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676109615612] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676109615612,5,FailOnTimeoutGroup] 2023-02-11 10:00:55,294 INFO [M:0;jenkins-hbase4:42723] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 10:00:55,294 INFO [M:0;jenkins-hbase4:42723] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 10:00:55,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676109615608] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676109615608,5,FailOnTimeoutGroup] 2023-02-11 10:00:55,295 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 10:00:55,295 INFO [M:0;jenkins-hbase4:42723] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 10:00:55,295 DEBUG [M:0;jenkins-hbase4:42723] master.HMaster(1502): Stopping service threads 2023-02-11 10:00:55,296 INFO [M:0;jenkins-hbase4:42723] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 10:00:55,296 ERROR [M:0;jenkins-hbase4:42723] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-11 10:00:55,297 INFO [M:0;jenkins-hbase4:42723] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 10:00:55,297 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 10:00:55,300 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 10:00:55,300 DEBUG [M:0;jenkins-hbase4:42723] zookeeper.ZKUtil(398): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 10:00:55,300 WARN [M:0;jenkins-hbase4:42723] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 10:00:55,300 INFO [M:0;jenkins-hbase4:42723] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 10:00:55,300 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 10:00:55,300 INFO [M:0;jenkins-hbase4:42723] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 10:00:55,301 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 10:00:55,301 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:55,301 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:55,301 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 10:00:55,301 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:55,301 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=843.28 KB heapSize=1.01 MB 2023-02-11 10:00:55,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 10:00:55,357 INFO [M:0;jenkins-hbase4:42723] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=843.28 KB at sequenceid=2048 (bloomFilter=true), to=hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1ac40b83c89f49d78e868815d42de2ef 2023-02-11 10:00:55,361 INFO [M:0;jenkins-hbase4:42723] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1ac40b83c89f49d78e868815d42de2ef 2023-02-11 10:00:55,361 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1ac40b83c89f49d78e868815d42de2ef as hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1ac40b83c89f49d78e868815d42de2ef 2023-02-11 10:00:55,365 INFO [M:0;jenkins-hbase4:42723] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1ac40b83c89f49d78e868815d42de2ef 2023-02-11 10:00:55,365 INFO [M:0;jenkins-hbase4:42723] regionserver.HStore(1080): Added hdfs://localhost:40183/user/jenkins/test-data/ede6ea3e-060e-7eb1-5bef-2688aa0bd9e7/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1ac40b83c89f49d78e868815d42de2ef, entries=282, sequenceid=2048, filesize=30.7 K 2023-02-11 10:00:55,366 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegion(2947): Finished flush of dataSize ~843.28 KB/863520, heapSize ~1.01 MB/1055160, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 64ms, sequenceid=2048, compaction requested=false 2023-02-11 10:00:55,370 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 10:00:55,370 DEBUG [M:0;jenkins-hbase4:42723] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676109655301Disabling compacts and flushes for region at 1676109655301Disabling writes for close at 1676109655301Obtaining lock to block concurrent updates at 1676109655301Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676109655301Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=863520, getHeapSize=1055160, getOffHeapSize=0, getCellsCount=2535 at 1676109655301Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676109655302 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676109655302Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676109655318 (+16 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676109655318Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676109655362 (+44 ms)Finished flush of dataSize ~843.28 KB/863520, heapSize ~1.01 MB/1055160, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 64ms, sequenceid=2048, compaction requested=false at 1676109655366 (+4 ms)Writing region close event to WAL at 1676109655370 (+4 ms)Closed at 1676109655370 2023-02-11 10:00:55,371 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,371 INFO [RS:2;jenkins-hbase4:35443] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35443,1676109614327; zookeeper connection closed. 2023-02-11 10:00:55,371 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@18c20755 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,371 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:35443-0x10042d2edb60003, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,371 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f22c3e1 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,376 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@30c6b34e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@30c6b34e 2023-02-11 10:00:55,381 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 10:00:55,381 INFO [M:0;jenkins-hbase4:42723] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 10:00:55,382 INFO [M:0;jenkins-hbase4:42723] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42723 2023-02-11 10:00:55,383 DEBUG [M:0;jenkins-hbase4:42723] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42723,1676109612965 already deleted, retry=false 2023-02-11 10:00:55,471 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,471 INFO [RS:1;jenkins-hbase4:41567] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41567,1676109614293; zookeeper connection closed. 2023-02-11 10:00:55,471 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a094b61 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,472 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): regionserver:41567-0x10042d2edb60002, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,472 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@727e2712 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,472 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@10298edf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@10298edf 2023-02-11 10:00:55,472 INFO [Listener at localhost/42305] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 10:00:55,572 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,572 INFO [M:0;jenkins-hbase4:42723] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42723,1676109612965; zookeeper connection closed. 2023-02-11 10:00:55,572 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d7d702c rejected from java.util.concurrent.ThreadPoolExecutor@392baa6a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,572 DEBUG [Listener at localhost/42305-EventThread] zookeeper.ZKWatcher(600): master:42723-0x10042d2edb60000, quorum=127.0.0.1:64863, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 10:00:55,572 ERROR [Listener at localhost/42305-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@90504 rejected from java.util.concurrent.ThreadPoolExecutor@392baa6a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 10:00:55,574 WARN [Listener at localhost/42305] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 10:00:55,586 INFO [Listener at localhost/42305] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 10:00:55,691 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 10:00:55,691 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1585950105-172.31.14.131-1676109609623 (Datanode Uuid 1b1dcb87-4b37-427d-8dc6-8c93c312eb72) service to localhost/127.0.0.1:40183 2023-02-11 10:00:55,693 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data5/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,693 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data6/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,695 WARN [Listener at localhost/42305] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 10:00:55,696 INFO [Listener at localhost/42305] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 10:00:55,802 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 10:00:55,802 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1585950105-172.31.14.131-1676109609623 (Datanode Uuid c54c5566-4672-48cb-89c2-ee4576f23616) service to localhost/127.0.0.1:40183 2023-02-11 10:00:55,803 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data3/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,803 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data4/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,804 WARN [Listener at localhost/42305] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 10:00:55,806 INFO [Listener at localhost/42305] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 10:00:55,911 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 10:00:55,911 WARN [BP-1585950105-172.31.14.131-1676109609623 heartbeating to localhost/127.0.0.1:40183] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1585950105-172.31.14.131-1676109609623 (Datanode Uuid e84cdfd9-aa67-445a-9a90-f8380b7333c7) service to localhost/127.0.0.1:40183 2023-02-11 10:00:55,912 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data1/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,912 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa32a73b-5029-d452-11f6-a4a0eaee957b/cluster_44f668ca-e1e0-159c-3cae-f75c3cbc9023/dfs/data/data2/current/BP-1585950105-172.31.14.131-1676109609623] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 10:00:55,961 INFO [Listener at localhost/42305] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 10:00:56,090 INFO [Listener at localhost/42305] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 10:00:56,180 INFO [Listener at localhost/42305] hbase.HBaseTestingUtility(1293): Minicluster is down